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ABSTRACT 


The  cost  and  effectiveness  of  computer-based  instruction 
for  military  training  are  evaluated  on  the  basis  of  about  30 
studies  conducted  since  1968.  Pour  methods  of  instruction  are 
distinguished  and  compared: 

Conventional  Instruction:  group-paced  lectures,  and 
discussions . 

Individualized  Instruction:  self-paced  (without  computer 
support ) . 

Computer-Assisted  Instruction  (CM):  computer  stores  and 
provides  instructional  materials  to  students  individually 
via  interactive  terminals;  computer  tests  and  guides 
students;  self-paced. 

Computer-Managed  Instruction  (CMI) : instructional  materials 
and  tests  provided  away  from  computer;  computer  scores 
the  tests  and  guides  students;  self-paced. 

Much  of  the  data  come  from  experiments  of  limited  duration  and 
with  relatively  few  students;  by  contrast,  some  CMI  systems  have 
been  used  for  4 years.  All  findings  are  confounded  by  effects 
that  may  be  due  either  to  CAI  or  CMI,  in  comparison  to  conven- 
tional instruction,  or  to  the  revisions  in  course  materials 
needed  to  modify  a course  from  conventional  to  CAI  or  CMI  instruc- 
tion. 

CAI  and  CMI  save  about  30  percent  (median)  of  the  time  re- 
quired by  students  to  complete  the  same  courses  given  by  conven- 
tional instruction;  CAI  and  CMI  cannot  be  compared  directly 
because  different  courses  were  used  in  each  study.  Student  at- 
trition appears  to  increase  with  CAI  and  CMI  compared  with 
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conventional  instruction,  but  changes  in  student  quality  may  also 
account  for  this  increase.  Students  prefer  CAI  or  CMI  to  con- 
ventional instruction;  attitudes  of  instructors,  considered  in 
only  a few  studies,  are  unfavorable  to  CAI  and  CMI.  Individual- 
ized instruction  (without  computer  support)  also  saves  student 
time;  little  additional  student  time  is  saved  when  the  same 
courses  are  given  by  CAI  or  CMI. 

Direct  comparisons  of  the  cost  and  effectiveness  of  differ- 
ent methods  of  instruction  are  not  now  possible  because  only 
incomplete  cost  data  were  found.  So-called  cost  savings  attrib- 
uted to  CAI  and  CMI  are  based  on  estimates  of  pay  and  allowances 
of  students  for  the  time  saved  by  these  methods  of  Instruction; 
allowances  are  seldom  made  for  the  costs  of  the  CAI  or  CMI  equip- 
ment and  courseware,  instructors,  and  other  costs  incremental  to 
computer-based  instruction. 
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SUMMARY 


This  paper  evaluates  the  cost  and  effectiveness  of  computer- 
assisted  and  computer-managed  instruction  for  use  in  military 
training.  The  military  Services  have  supported  research  and 
development  on  these  methods  of  instruction  since  about  I960. 

A.  METHODS  OF  INSTRUCTION 

Methods  of  instruction  can  be  placed  conveniently  in  four 
groups,  as  described  below: 

• Conventional  instruction,  where  an  instructor  may  use 
lectures,  discussions,  laboratory  demonstrations,  and 
tutorial  sessions.  Groups  of  students  proceed  through 
the  curriculum  at  the  same  pace;  differences  in  achieve- 
ment among  students  are  reflected  in  grades  at  the  end 
of  the  course. 

• Individualized  Instruction,  where  each  student  proceeds 
at  his  own  pace  through  the  curriculum  that  is  arranged 
in  a series  of  lessons  and  tests.  Mastery  of  each  lesson 
is  set  as  a condition  of  progress.  Differences  among 
students  are  reflected  in  the  amounts  of  time  needed  to 
complete  the  course,  although  grades  may  also  be  given. 

In  general,  an  effort  is  made  to  assure  about  the  same 
level  of  achievement  for  all  students. 

• Computer-assisted  Instruction  (CAI),  where  all  instruc- 
tional materials,  i.e.,  lessons  and  tests,  are  stored  in 
the  computer;  the  student  interacts  with  this  material 
in  real  time  via  a terminal  and  display  system.  The 
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computer  can  perform  many  functions,  such  as  diagnose 
student  performance,  prescribe  lessons,  maintain  records 
on  student  progress,  and  predict  individual  course  comple- 
tion dates.  Current  CAI  systems  differ  in  the  number 
of  terminals  linked  to  a central  computer  (1  to  1000) 
and  location  of  the  central  computer  (which  may  require 
long-distance  communications).  In  "stand-alone"  sys- 
tems, a terminal  and  its  computer  comprise  the  entire 
system.  The  PLATO  IV  system  is  used  in  courses  for 
medical  technicians  at  Sheppard  APB  and  for  vehicle 
repair  at  Chanute  AFB.  TICCIT  is  used  in  courses  for 
tactical  coordinators  for  S-3A  aircraft  at  Naval  Air 
Station,  North  Island.  GETS,  a stand-alone  system,  will 
be  used  to  handle  training  overloads  in  the  TRIDENT 
program. 

• Computer-managed  instruction  (CMI).  where  instruction 
using  self-paced  lessons  takes  place  away  from  the 
computer.  The  computer  scores  the  tests  and  interprets 
results  to  each  student;  advises  him  to  take  following  or 
alternative  lessons;  recommends  remediation;  and  manages 
student  records,  instructional  resources,  and  administra- 
tive data.  The  Air  Force  Advanced  Instructional  System 
( AIS ) is  a prototype  CMI  system  used  for  technical  train- 
ing at  Lowry  AFB.  It  can  support  up  to  3000  students  a 
day  in  four  courses;  the  present  version  consists  of  50 
student  terminals  (for  scoring  tests),  11  management 
terminals  (for  use  by  instructors),  and  a CDC  CYBER  73-16 
computer.  The  Navy  Computer  Managed  Instruction  System 
(Navy  CMI)  at  Naval  Air  Technical  Training  Center, 
Millington,  Tennessee,  now  handles  about  6000  students  a 
day  in  11  schools  at  five  training  centers  in  the  United 
States;  by  1980,  it  is  expected  to  handle  16,000  students 
in  2k  schools  at  6 centers. 
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B.  APPLICATIONS  OF  CAI  AND  CHI  IN  MILITARY  TRAINING 

CAI  and  CMI  seem  well-suited  to  providing  specialized  skill 
training  both  at  military  schools  and  at  operational  units  in 
the  field.  Skill  training  at  military  schools  is  estimated  to 
cost  $3  billion  a year  and  produce  1.1-million  course  graduates 
a year  (PY  1979  data).  The  amount  of  technical  training  that 
occurs  in  operational  units,  r.e.,  away  from  formal  schools,  is 
thought  to  be  large,  but  its  magnitude  is  unknown;  this  includes 
on-the-job  training,  crew  and  unit  training,  refresher  and  up- 
grade training. 

The  Department  of  Defense  is  estimated  to  spend  about  $12 
million  a year  for  research  and  development  on  the  use  of  com- 
puters in  military  education  and  training  (FY  1977  data). 

C.  NATURE  OF  THE  DATA  ON  COST  AND  EFFECTIVENESS  OF  CAI  AND  CMI 

The  use  of  computer-assisted  and  computer-managed  instruc- 
tion In  military  training  has  been  evaluated  in  about  30  studies 
(producing  48  sets  of  data)  since  1968.  Most  (70  percent)  of 
the  data  on  CAI  come  from  experiments  with  few  students  (up  to 
50)  and  limited  course  materials  (1  day  to  1 week).  There  are 
fewer  studies  of  CMI  but  these  involve  more  students  (600  to  2500) 
and  longer  courses  (2  to  10  months).  There  is  a wide  range  of 
subject  matter  in  these  studies,  e.g.t  knowledge,  theory,  and 
hands-on  performance  skills;  electronics  machinist,  recipe  con- 
version, vehicle  repair,  fire-control  technician. 

Each  of  the  30  studies  report  effectiveness.  However,  only 
eight  of  the  studies  which  report  effectiveness  also  provide 
some  cost  data.  The  latter  data  are  limited  to  expenses  incurred 
during  the  experiment  and  are  incomplete  with  respect  to  costs  of 
program  management,  maintenance  and  repair,  instructional  support, 
and  other  factors  important  in  determining  life-cycle  costs.  It 
is  probably  inappropriate  to  extrapolate  from  cost  data  in  experi- 
ments to  the  costs  of  large-scale,  long-term  operational  training 
programs . 
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The  effectiveness  of  training  should  be  measured  by  how 
well  course  graduates  perform  specific  Jobs  in  operational  units. 
Instead,  all  studies  use  student  achievement  at  school  as  a 
measure  of  effectiveness.  The  relation  between  achievement  at 
school  and  performance  on  the  Job  is  essentially  unknown,  even 
for  conventional  instruction.  Data  on  length  of  time  required 
for  students  to  complete  a course  (generally  less  for  CAI  and  CMI 
than  for  conventional  instruction)  should  be  treated  as  a measure 
of  the  cost  of  instruction  rather  than  a measure  of  its  effective 
ness.  The  same  argument  applies  to  academic  attrition  rate. 

The  attitudes  of  students  and  instructors  to  CAI  and  CMI  may  be 
interesting;  however,  they  are  qualitative  in  nature  and  it  is 
difficult  to  relate  such  data  either  to  the  cost  or  the  effective 
ness  of  instruction. 

The  comparisons  of  alternative  methods  of  instruction  are 
limited.  Generally,  CAI  or  CMI  is  compared  to  conventional 
instruction;  we  found  only  a few  comparisons  of  CAI  and  CMI  with 
individualized  instruction  (without  computer  support),  a compari- 
son which  relates  to  the  benefits  of  computer  support.  In  addi- 
tion, time  savings  found  when  CAI  or  CMI  are  compared  to  con- 
ventional instruction  may  be  due  to  a combination  of  self-pacing, 
computer  support,  revised  and  possibly  reduced  amounts  of  course 
materials . 

D.  EFFECTIVENESS  OF  CAI  AND  CMI 

Based  on  evidence  provided  by  military  research  studies 
and  qualified  as  noted  above,  the  effectiveness  of  CAI  and  CMI 
is  evaluated  as  follows: 

• Student  achievement.  Student  achievement  at  school  with 
CAI  is  about  the  same  as  that  with  conventional  instruc- 
tion in  most  comparisons  and  superior  in  about  one-third 
of  the . comparisons . The  differences  in  achievement  are 
not  thought  to  have  practical  significance.  Student 


achievement  with  CMI  is  about  the  same  as  that  with  con- 
ventional instruction.  These  findings  are  important  but 
also  inevitable  because  students  are  held  in  CAI  and  CMI 
courses  until  they  achieve  at  least  the  standards  estab- 
lished previously  for  conventional  instruction. 

• Student  time  savings.  Students  instructed  by  CAI  or  CMI 
save  about  30  percent  (median  value)  of  the  time  required 
to  complete  the  same  courses  given  by  conventional  in- 
struction. There  is  a wide  range  in  amounts  of  time 
reported  as  saved  in  these  studies.  The  amounts  of  time 
saved  by  CAI  and  CMI  cannot  be  compared  directly  because 
different  courses  were  used  for  tests  of  these  methods 

of  instruction.  Where  courses  have  been  given  for 
relatively  long  times,  the  initial  student  time  savings 
are  maintained  and,  despite  monthly  fluctue  ions,  tend 
to  increase.  This  finding  is  based  on  four  courses  given 
by  the  Air  Force  Advanced  Instructional  System  for  about 
4 years  and  on  three  courses  given  by  the  Navy  Computer 
Managed  Instruction  System  for  about  15  months;  both 
systems  are  CMI  systems. 

• Student  attrition . The  academic  elimination  rates  in 
four  courses  on  the  Air  Force  Advanced  Instructional 
System  (AIS)  appear  to  have  increased  slightly  over  4 
years  compared  to  the  previous  base  rates;  however,  the 
average  academic  elimination  rate  for  all  courses  at 
Lowry  AFB,  i.e.,  those  not  on  AIS,  increased  at  the 
same  time.  Thus,  the  increase  in  attrition  may  be  at- 
tributed to  AIS  {i.e.,  CMI)  instruction  or  to  a decrease 
in  student  quality  or  to  some  combination  of  these  two 
factors.  Similar  increases  in  attrition  seem  to  have 
occurred  in  six  courses  on  the  Navy  CMI  system  over  a 
15-month  period;  attrition  dropped  in  one  course;  data 
on  non-CMI  courses  for  the  same  time  period  were  not 
provided. 
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• Attitudes  of  students  and  Instructors.  Students  in 
experiments  almost  always  prefer  CAI  or  CMI  to  conven- 
tional instruction.  The  attitudes  of  instructors  are 
reported  only  in  a few  studies  but  these  are  almost 
always  unfavorable  to  CAI  and  CMI  in  comparison  to  con- 
ventional Instruction. 

• Time  savings  found  with  Individualized  instruction  and 
computer-based  Instruction.  Some  data  were  found  where 
the  same  course  was  given  by  conventional  instruction, 
individualized  instruction  (£.«.,  self-paced  instruction 
without  computer  support)  and  either  CAI  or  CMI.  Indi- 
vidualized instruction  saves  student  time.  However, 

the  addition  of  computer  support  (either  CAI  or  CMI)  to 
individualized  instruction  does  not  increase  the  amount 
of  student  time  saved  very  much  beyond  that  achieved  by 
individualized  instruction  alone  (i.e.,  without  computer 
support) . Again,  differences  between  time  savings  attrib- 
uted to  CAI  and  CMI  cannot  be  evaluated  because  different 
courses  were  used  in  each  group  of  studies.  These  data 
do  not  necessarily  imply  that  the  addition  of  CAI  or  CMI 
to  individualized  Instruction  (i.e.,  transforming  the 
method  of  Instruction)  Is  not  cost-effective.  That  would 
depend  on  whether  the  incremental  costs  of  computer  sup- 
port are  offset  by  benefits  in  other  areas  such  as,  e.g 
a need  for  fewer  instructors  and  support  personnel  and 
for  less  administrative  support. 

E.  COSTS  OF  INSTRUCTION  IN  MILITARY  TRAINING 

The  benefits  of  computer-based  instruction  have  to  be 
compared  with  the  cost  of  providing  this  type  of  instruction, 
but  only  incomplete  cost  data  were  found. 

• Collection  of  cost  data.  The  military  Services  maintain 
systems  that  report  the  costs  of  individual  courses. 
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These  are  useful  for  such  purposes  as  setting  reimburse- 
ment rates  for  training  students  from  other  Services  or 
other  governments.  They  are  not  useful  for  analyses  of 
the  costs  of  different  methods  of  instruction  for  the 
following  reasons:  (1)  they  do  not  distinguish  the 
costs  of  parts  of  a course,  which  would  permit  determin- 
ing the  costs  of  different  methods  of  instruction  used 
within  a course;  (2)  costs  of  training  support  and 
management,  that  may  vary  considerably  between  methods 
of  instruction,  are  allocated  to  individual  courses  on 
essentially  arbitrary  bases,  such  as  the  student  load 
of  all  courses . 

• Type  of  data  needed  on  cost  of  instruction.  Each  method 
of  instruction  in  military  training  requires  the  expendi- 
ture of  funds  for  most,  but  not  necessarily  all,  of  the 
following  functions: 

PROGRAM  DEVELOPMENT 

Program  Design 

Instructional  Materials 

Conventional  Instruction 
Individualized  Instruction 

Programming 
First-Unit  Production 

Computer-Based  Instruction 

Programming 

Coding 

PROGRAM  DELIVERY 
Instruction 
Instructors 

Instructional  Support  Personnel 

Equipment  and  Services 

Laboratory  (including  simulators) 

Media  Devices 
Computer  Systems 
Communications 
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Materials  (including  Consumables) 

Facilities 

PROGRAM  MANAGEMENT  AND  ADMINISTRATION 

STUDENT  PERSONNEL 

Pay  and  Allowances 

Other  (Permanent  Change  of  Station, 

Temporary  Duty) 

Limited  cost  data  were  found  for  some  of  these  resources 
and  these  are  presented  in  the  report.  Cost  data  were 
net  found  or  were  extremely  limited  for  the  following 
resources  for  all  methods  of  instruction: 

- Program  Design 

- Instructional  Material:  conventional  instruction 

- Instructional  Support  Personnel 

- Laboratory  Equipment 

- Materials  (including  consumables) 

- Program  Management  and  Administration 

- Student  Personnel:  Permanent  Change  of  Station, 

Temporary  Duty,  etc. 

• Collection  of  More  Complete  Data.  Detailed  cost  data, 
required  for  analytical  purposes,  may  be  collected  in 
three  possible  ways: 

- Universal,  more  complete  reporting  for  all  courses 
and  support  functions 

- Sampling  selected  courses  and  support  functions 

- Ad  hoc 

The  costs  and  benefits  of  these  ways  of  collecting  the  cost 
data  needed  to  evaluate  alternative  methods  of  instruction  should 
be  examined. 

F.  COST-EFFECTIVENESS  OF  COMPUTER-BASED  INSTRUCTION 

There  have  been  few  attempts  to  assess  the  cost-effectiveness 
of  computer-assisted  or  computer-managed  instruction  and  all  of 
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these  are  limited,  as  indicated  above,  particularly  with  respect 
to  the  cost  data  that  have  been  used.  The  following  results 
have  been  reported: 

• CAI . The  PLATO  IV  system  was  Judged  to  be  not  cost- 
effective  in  two  evaluations.  Although  substantial 
amounts  of  student  time  were  saved  (19  to  89  percent  in 
eight  courses),  PLATO  IV  was  Judged  to  be  not  as  cost- 
effective  as  self-paced  instruction  (because  of  high 
communications  and  maintenance  costs)  in  one  case  and 
not  as  cost-effective  as  programmed  instruction  (because 
of  greater  development  and  operating  costs)  in  the 
second  case. 

• CMI . It  was  estimated  that  the  Navy  CMI  system  avoided 
costs  of  $10  million  in  FY  1977  and  that  the  Air  Force 
AIS  avoided  costs  of  $3  million  in  FY  1978.  Both  of 
these  estimates  are  derived  by  translating  amounts  of 
student  time  saved  into  dollars  avoided  for  student  pay 
and  allowances  because  of  the  reduced  training  times. 

The  costs  of  providing  CMI  instruction  are  not  considered 
in  these  reports.  In  a recent  test,  the  AIS  was  Judged 
to  be  cost-effective,  compared  to  instructor-supported 
self-pacing  in  one  course  but  not  in  three  others  because 
of  costs  attributed  to  the  AIS  computer;  however,  the 
computer  costs  were  small  in  comparison  to  other  school 
costs . 
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CONCLUSIONS 


A.  MEASURES  OF  EFFECTIVENESS 

The  effectiveness  of  computer-assisted  and  computer-managed 
instruction  for  military  training  has  been  measured  only  by 
student  achievement  at  school  and  not  by  performance  on  the  Job. 
Correlations  between  performance  at  school  and  on  the  Job  have 
not  been  established  for  any  method  of  instruction. 

B.  STUDENT  ACHIEVEMENT  AT  SCHOOL 

Student  achievement  in  courses  at  military  training  schools 
with  computer-assisted  instruction  is  the  same  as  or  greater  than 
that  with  conventional  instruction;  the  amount  of  additional 
achievement  is  small  and  has  little  practical  importance.  Student 
achievement  in  courses  with  computer-managed  instruction  is  about 
the  same  as  that  with  conventional  instruction.  Both  of  these 
results  are  due  to  keeping  students  in  CAI  and  CMI  courses  until 
they  achieve  standards  set  previously  for  conventional  instruc- 
tion . 

C.  STUDENT  TIME  SAVINGS 

Computer-assisted  and  computer-managed  instruction  in  mili- 
tary training  save  about  30  percent  of  the  time  (median  value) 
needed  by  students  to  complete  the  same  courses  given  by  con- 
ventional instruction.  The  amounts  of  time  reported  as  saved 
vary  widely,  but  little  attention  has  been  given  to  the  factors 
that  could  account  for  the  wide  variation.  Most  of  the  results 
on  computer-assisted  instruction  come  from  experiments  of  limited 
duration,  with  limited  amounts  of  course  materials,  and  with 

11 


relatively  few  students.  Where  computer-managed  instruction  has 
been  used  for  extended  periods  (up  to  4 years),  the  initial  time 
savings  have  been  maintained  or  increased. 

0.  INDIVIDUALIZED  AND  COMPUTER-BASED  INSTRUCTION 

Individualized  instruction  (self-paced  instruction  without 
computer-support)  saves  student  time;  little  or  no  additional 
student  time  is  saved  when  the  same  courses  are  given  by  computer- ' 
assisted  or  computer-managed  instruction. 

E.  STUDENT  ATTRITION 

Computer-managed  Instruction  may  increase  the  rate  of  student 
attrition  for  academic  reasons,  compared  to  that  with  conventional 
instruction.  The  observed  increases  in  attrition  may  also  be  due, 
at  least  in  part,  to  decreases  in  student  quality,  but  this 
relationship  has  not  been  carefully  examined.  Student  attrition 
appears  not  to  Increase  with  computer-assisted  instruction,  but 
this  finding  is  based  on  tests  of  limited  duration. 

F.  STUDENT  AND  INSTRUCTOR  ATTITUDES 

Attitudes  of  students  toward  computer-assisted  and  computer- 
managed  instruction  appear  to  be  favorable.  Attitudes  of  in- 
structors  are  reported  as  unfavorable,  but  this  finding  is  based 
on  very  limited  data.  Little  attention  has  been  given  to  the 
role  of  instructors  in  computer-based  Instruction  and  to  how  they 
should  be  prepared  for  this  type  of  instruction. 

G.  COST  DATA 

Only  limited  and  incomplete  data  are  available  on  the  costs 
of  computer-assisted  and  computer-managed  instruction  in  military 
training.  Data  that  are  collected  routinely  on  the  costs  of 
operational  training  programs  are  too  highly  aggregated,  partic- 
ularly with  respect  to  training  support  functions,  for  use  in 
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analytical  comparisons  of  computer-based  instruction  with  con- 
ventional instruction. 

H.  COST-EFFECTIVENESS 

Estimates  based  on  the  amounts  of  student  time  saved  sug- 
gest that  the  Navy  Computer  Managed  Instruction  System  avoided 
costs  of  about  $10  million  in  FY  1977  and  that  the  Air  Force 
Advanced  Instructional  System  avoided  costs  of  about  $3  million 
in  FY  1978.  These  estimates  are  incomplete  because  they  do  not 
consider  the  other  costs  of  providing  computer-managed  instruc- 
tion at  these  installations  or  compare  these  costs  with  the  costs 
of  alternative  methods  of  instruction  for  the  same  courses. 
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RECOMMENDATIONS 


A.  JOB-PERFORMANCE  DATA 

Improve  methods  currently  available  for  measuring  perform- 
ance on  the  Job  In  areas  related  to  technical  training.  Compare 
achievement  at  school  with  performance  on  the  Job  for  students 
in  courses  given  by  computer-assisted  and  computer-managed  in- 
struction; to  whatever  extent  opportunities  exist,  do  the  same 
thing  for  the  same  courses  given  by  conventional  and  individual- 
ized instruction.  The  Job-performance  data  should  be  collected 
for  several  time  intervals  after  students  leave  school  to  deter- 
mine whether  benefits  in  favor  of  any  method  of  instruction  are 
sustained  as  Job  experience  increases. 

B.  COST  AND  EFr::C'i  1 VENESS  DATA 

Evaluate  alternative  methods  of  collecting  reliable  data  on 
the  costs  and  effectiveness  of  instruction  in  military  training. 
Based  on  these  findings,  develop  and  initiate  data-collection 
programs  on  the  costs  and  effectiveness  of  alternative  methods 
of  instruction. 

C.  RESEARCH  AND  DEVELOPMENT  PLAN 

Bring  up  to  date  the  "Integrated  Department  of  Defense  Plan 
for  Research  and  Development  on  Computers  in  Education  and  Train- 
ing" (Department  of  Defense,  September  1975)-  Support  is  needed 
for  Exploratory  and  Advanced  Development  (6.2  and  6.3  RDT&E 
funds)  on  many  subjects  identified  in  this  paper,  such  as  the 
development  of  objective  measures  of  performance  on  the  job, 
comparisons  of  student  achievement  at  school  with  performance  on 
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the  job,  the  development  of  methods  to  measure  the  quality  of 
course  materials  and  delivery  of  instruction,  and  studies  to 
account  for  the  relative  contributions  of  self-pacing,  course 
revision,  computer  support,  and  other  factors  to  the  amounts  of 
student  time  saved  by  computer-assisted  and  computer-managed 
instruction.  Support  for  other  studies  to  improve  various 
aspects  of  computer-assisted  and  computer-managed  instruction 
may  well  be  questioned  until  more  reliable  cost  data  are  avail- 
able to  determine  areas  of  high  pay-off. 

D.  CURRENT  ANO  PROPOSED  SYSTEMS 

Collect  data  on  the  costs  of  instruction  for  courses  and 
course  segments  given  now  by  computer-assisted  or  computer- 
managed  instruction  for  military  training,  e,g.t  PLATO  IV  at 
Sheppard  Air  Force  Base,  Texas,  and  at  Chanute  Air  Force  Base, 
Illinois;  TICCIT  at  North  Island  Naval  Air  Station,  San  Diego, 
California;  Advanced  Instructional  System  at  Lowry  Air  Force 
Base,  Denver,  Colorado;  and  Navy  Computer  Managed  Instruction 
System  at  Naval  Air  Technical  Training  Center,  Millington, 
Tennessee.  Comparable  baseline  cost  data  should  also  be  col- 
lected, as  far  as  possible,  for  alternative  methods  of  instruc- 
tion for  the  same  courses.  Projections  of  cost  should  be  made 
for  computer-managed  Instruction  systems  that  are  now  being 
planned;  i.e.  , the  Navy  Aviation  Training  Support  System,  the 
Army  Automated  Instructional  Management  System,  and  the  Marine 
Corps  Communication-Electronics  School  CAI/CMI  System. 

E.  RANGE  OF  TIME  SAVINGS 

Determine  the  factors  which  account  for  the  large  variations 
in  the  amounts  of  student  time  saved  by  computer-assisted  and 
computer-managed  instruction  in  various  studies.  Consideration 
should  be  given  to  such  factors  as  quality  of  courseware  (in- 
cluding that  in  conventional  courses),  instructional  strategy. 
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types  of  subject  matter  presented  in  courses,  and  the  amount  and 
type  of  guidance  provided  by  instructors.  An  effort  should  also 
be  made  to  resolve  the  extent  to  which  such  factors  as  self- 
pacing, course  revision,  shortening  courses,  and  various  types 
of  computer-support  contribute  to  the  total  amounts  of  student 
time  saved. 

F.  STUDENT  ATTRITION 

Determine  the  extent  to  which  observed  increases  of  student 
attrition  with  computer-managed  instruction  are  due  to  this  method 
of  instruction  and  to  other  factors  that  may  also  be  present , 
such  as  changes  in  the  quality  of  students. 

G.  INSTRUCTORS'  ATTITUDES 

Determine  the  attitudes  of  instructors  to  computer-based 
and  other  methods  of  instruction  in  a systematic  manner  so  that 
remedial  actions  can  be  taken  as  required. 


I.  INTRODUCTION 


A.  PURPOSE 

The  purpose  of  this  study  Is  to  evaluate  research  and  devel- 
opment on  the  cost-effectiveness  of  computer-based  instruction 
for  military  training. 

The  use  of  computers  to  provide  and  support  instruction  is 
the  result  of  significant  developments  that  have  occurred  since 
about  I960:  (1)  growth  in  the  capabilities  of  computer  hardware 

and  software  and  (2)  improved  procedures  for  designing  lessons 
in  a self-paced  or  individualized  format  needed  for  computer- 
based  instruction.  The  Department  of  Defense  and  the  military 
Services  have  supported  the  development  of  computer-based  instruc- 
tion because  of  its  potential  value  to  improve  the  effectiveness 
and  reduce  the  cost  of  training,  particularly  where  large  numbers 
of  students  are  involved. 

This  study  was  performed  for  the  Deputy  Under  Secretary  of 
Defense  for  Research  and  Engineering  (Research  and  Advanced  Tech- 
nology) , Office  of  the  Under  Secretary  of  Defense  for  Research 
and  Engineering.  It  responds  to  a recommendation  made  by  the 
Defense  Science  Board:* 

To  improve  the  effectiveness  of  training  and 

training  technology  R&D,  the  DoD  should: 

1.  Develop  a capability  to  perform  cost- 
effectiveness  analyses  of  training 
technology . 


•Summary  Report  of  the  Task  Force  on  Training  Technology, 
Defense  Science  Board,  27  February  1976,  (p.  x). 
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B.  METHODS  OF  INSTRUCTION 


The  Rand  Corporation's  "Method  of  Designing  Instructional  j 

Alternatives  (MODIA)"  identifies  20  different  methods  of  teach-  ’ 

i 

ing  (Carpenter-Huffman,  1977).  For  convenience,  methods  of  in-  j 

struction  are  organized  here  in  four  groups;  more  than  one  method 
of  instructidn  may  be  used  in  a course.  (See  Appendix  A for  a 
more  complete  discussion.) 

1 . Conventional  Instruction  ' 

! 

Conventional  instruction  refers  to  many  possible  combina-  ' 1 

tions  of  lectures,  discussions,  laboratory,  and  tutorial  sessions  ■ j 

as  a method  of  instruction.  A key  feature  of  conventional  in-  ■.  I 

struction  is  that  groups  of  students  proceed  through  a course  at 
the  same  pace.  Differences  in  the  amount  of  information  retained 
by  students  are  reflected  in  their  grades  at  the  end  of  the  i 

course.  Conventional  instruction  is  used  in  75  to  90  percent  of 
all  military  courses,  although  a precise  estimate  is  not  avail- 
able. It  is  also  referred  to  as  lock-step  Instruction,  platform  •] 

instruction,  and  group  scheduling. 

■ •] 

2.  Individualized  Instruction  i 

— • 

In  individualized  instruction,  a course  is  arranged  in  a j 

series  of  lessons  and  tests  and  each  student  proceeds  at  his  own 
pace.  Mastery  of  each  lesson  is  prescribed  as  a condition  of 
progress.  Differences  among  students  are  reflected  in  how  long 
it  takes  them  to  complete  a course,  although  grades  may  also  be 
given. 

There  are  various  forms  of  Individualized  instruction  that 
differ  primarily  in  such  ways  as  the  structure  of  lessons  pro- 
vide! to  the  student  (main  line,  branching)  and  the  extent  to 
•,<nlch  the  student  is  completely  ^ree  to  proceed  at  his  own  pace. 

All  methods  of  computer-based  instruction  rely  on  some  form  of 
Individualized  Instruction;  by  definition,  the  term  "individual- 
ized instruction"  will  be  used  here  to  apply  only  to  this  method 

I 

i 
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of  instruction  conducted  without  computer  support.  The  terms 
individualized  instruction,  self-pacing,  and  programmed  instruc- 
tion will  be  used  synonomously  unless  otherwise  specified. 

3.  Computer-Assisted  Instruction  (CAI) 

In  this  paper  the  term  QOmputsr-baeed  instruction  refers 
generally  to  both  CAI  and  CMI  methods  of  instruction.  In 
computer-assisted  instruction  (CAI),  the  student  interacts  in 
real  time,  via  an  Interactive  terminal,  with  instructional 
material  that  is  stored  in  the  computer.  Thi3  offers  great 
flexibility  for  presenting  alternative  versions  of  the  3ame  les- 
sons according  to  each  student's  particular  way  of  learning. 

Most  CAI  systems  diagnose  student  performance,  prescribe  les- 
sons, and  maintain  student  records.  Examples  of  some  CAI  systems 
follow  (see  Appendix  B for  a more  complete  discussion): 

• PLATO:  Programmed  Logic  for  Automatic  Teaching  Opera- 
tion. A current  version  of  this  system,  PLATO  IV,  can 
support  about  950  terminals  linked  through  microwave  and 
land-line  communications  to  a large  central  computer 

( CDC  CYBER  74)  located  at  the  University  of  Illinois. 

• TICCIT:  Time-Shared  Interactive  Computer-Controlled 
Information-Television.  The  basic  TICCIT  system  uses 
one  or  two  mini-computers  to  support  up  to  128  terminals 
at  one  location. 

• LTS : Lincoln  Terminal  System.  The  latest  version,  LTS-5, 
uses  microfiche  to  store  both  visual  images  and  an  audio 
track.  This  is  a self-contained  or  "stand-alone"  system. 

• GETS : General  Electric  Training  System.  This  is  a 
stand-alone  system  which  uses  a random  access  35-mm  slide 
projector  for  visual  displays  and  floppy  discs  for  les- 
son preparation  and  playback. 

4 . Computer-Managed  Instruction  (CMI) 

In  computer-managed  instruction  (CMI),  instruction  takes 
place  away  from  the  computer.  The  computer  scores  tests  and 
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interprets  results  to  the  student;  advises  on  following  or  alter- 
native lessons;  recommends  remediation;  and  manages  student  re- 
cords, resources,  and  administrative  data.  (See  Appendix  B for 
a more  complete  discussion.) 

This  process  is  initiated  typically  when  the  student  places 
a test  answer  sheet  on  an  optical  reader  connected  to  the  central 
computer.  He  receives  the  results  on  a printout  which  tells  him 
how  well  he  performed,  what  lesson  to  take  next,  and  where  to 
find  it.  Examples  of  some  CMI  systems  follow: 

• AIS : Advanced  Instructional  System.  This  prototype 
system  is  installed  at  the  Air  Force  Technical  Training 
Center,  Lowry  Air  Force  Base,  Denver,  Colorado.  The 
present  version  consists  of  50  student  terminals,  11 
management  terminals,  and  a CDC  CYBER  73-16  computer 
which  can  support  up  to  3,000  students  a day  in  four 
courses.  These  courses  were  selected  to  represent  a 
cross  section  of  the  technical  training  courses  at  Lowry 
AFB  and  serve  about  25  percent  of  the  student  body  there. 
The  management  terminals  provide  CAI  services  for  use  by 
instructors  (for  developing  or  revising  lessons  and  for 
retrieving  data  collected  by  the  system) . The  system 
could  be  expanded  to  provide  CAI  services  to  students . 

• Navy  CMI : Computer  Managed  Instruction  System.  This 
system,  installed  at  Naval  Air  Technical  Training  Center, 
Millington,  Tennessee  (also  referred  to  as  Memphis, 
Tennessee),  handles  about  6,000  students  in  11  schools 

at  5 centers.  It  is  based  on  a Honeywell  Series  60, 
level  66  computer. 

• CTS : Computerized  Training  System.  This  system  can 
provide  CAI  and  CMI  services  for  128  terminals  at  the 
U.S.  Army  Signal  Center  and  School,  Fort  Gordon,  Georgia. 
It  is  based  on  sik  mini-computers  (PDP-ll/35s ) . Each 
terminal  contains  a visual  display  unit  and  a keyboard 
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which  can  provide  both  interactive  instruction  and  course 
management  services.  (Note:  A report  evaluating  the  CTS 
in  a CMI  mode  arrived  too  late  for  use  in  this  paper. 

See  Seidel,  Rosenblatt,  Wagner,  Schulz,  and  Hunter, 

1978. ) 

C.  DISTINCTIONS  BETWEEN  MILITARY  AND  NON-MILITARY  INSTRUCTION 

Military  personnel  receive  pay  and  allowances  while  they  are 
in  training.  Thus,  any  procedure  which  can  reduce  the  len  th 
of  time  required  for  training,  without  significantly  affecting 
the  amount  and/or  quality  of  information  acquired,  can  assist  in 
reducing  the  cost  of  training  at  military  schools;  it  can  also 
result  in  increasing  the  amount  of  time  spent  by  military  person- 
nel in  operational  assignments  during  their  military  careers. 
Military  training  courses  are  designed  to  qualify  students  for 
well-defined  Jobs  to  which  they  can  be  assigned  upon  successful 
completion  of  these  courses. 

The  situation  differs  in  almost  all  tvpes  of  public  and 
private  education  where  students  remain  at  school  for  required 
periods  of  time  and  are  not  paid  while  being  Instructed.  These 
schools  receive  no  direct  benefits  for  completing  instruction  in 
less  than  the  required  time.  Courses  are  generally  not  designed 
to  qualify  students  for  particular  Jobs  and,  obviously,  schools 
cannot  assign  students  to  Jobs  when  they  graduate. 

A major  consequence  of  these  distinctions  is  that  methods 
of  instruction  that  are  cost-effective  for  military  training 
may  not  be  cost-effective  in  other  areas.  Another  is  that  re- 
search on  computer-based  instruction  supported  by  the  military 
Services  has  emphasized  the  possibility  of  saving  student  time 
while  maintaining  student  achievement  constant.  Research  on 
instruction  in  non-military  settings  has  been  concerned  more 
with  the  amount  of  student  achievement  at  the  completion  of  a 
course  than  with  the  amount  of  time  needed  by  students  to  acquire 
the  material. 
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0.  APPLICATION  OF  CAI-  AND  CMI 


The  potential  application  of  CAI  and  CMI  would  appear  to  be 
primarily  for  specialized  skill  training  at  technical  schools 
which  prepare  military  personnel  for  specific  jobs  in  the  mili- 
tary Services.  About  300,000  people  complete  recruit  training 
each  year  and  become  candidates  for  specialized  skill  training. 
Skill  training  is  estimated  to  cost  $3  billion  and  to  produce 
1.1  millior.  course  graduates  each  year  (1214,000  man-years  of 
training  in  FT  1979);  about  79,000  people  (75  percent  military) 
are  needed  to  conduct  and  support  this  training  (Department  of 
Defense,  Military  Manpower  Training  Report  for  FY  1979,  March 
1978).  About  75  percent  of  all  training  loads  are  for  new 
accessions  to  the  military  Services. 

CAI  and  CMI  also  appear  appropriate  for  certain  types  of 
training  that  occurs  away  from  formal  schools,  such  as  on-the-job 
training,  crew  and  unit  training,  refresher  and  upgrade  training 
in  .operational  units.  The  magnitude  of  these  efforts  is  thought 
to  be  large  but  no  estimate  of  its  cost  has  been  made.  The 
"Integrated  DoD  Plan  for  R&D  on  Computers  in  Education  and 
Training",  prepared  by  a tri-service  group  in  September  1975, 
proposed  that  $12.1  million  be  allocated  in  FY  1977  for  research 
and  development  on  computer-based  instruction.  An  estimate  of 
the  funds  allocated  by  the  DoD  to  R&D  on  computer-based  instruc- 
tion in  recent  years  has  not  been  made. 
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II.  PROBLEMS  ENCOUNTERED  IN  EVALUATING  THE  COST  EFFECTIVENESS 
GF  COMPUTER-ASSISTED  AND  COMPUTER-MANAGED  INSTRUCTION 


The  military  Services  have  provided  strong  support  to  re- 
search and  development  on  computer-based  instruction  since  the 
early  1960s  because  of  its  obvious  application  to  military  train- 
ing. This  same  time  period  saw  the  development  of  new  tools  for 
analysis  and  management  of  military  (and  other  government)  re- 
sources. The  analytic  procedures  are  best  described  as  the 
adaptation  of  traditional  economic  analysis  to  government  opera- 
tions . 

Traditional  economic  analysis  is  identified  with  production 
processes  in  which  organized  markets  exist  for  determining  the 
values  of  both  resource  inputs  and  outputs  in  a common  unit  of 
measure,  such  as  dollars.  In  military  activities,  resource  in- 
puts are  typically  obtained  from  organized  markets  and  valued  in 
dollars,  but  no  such  market  exists  for  determining  the  dollar 
value  of  resource  outputs  (e.y.,  the  cost  of  military  training 
may  be  determined,  but  what  dollar  value  should  be  placed  on  its 
results?).  The  lack  of  comparability  between  inputs  and  outputs 
in  economic  analyses  of  military  systems  has  led  to  the  develop- 
ment of  special  analytic  techniques.  Cost-effectiveness  analysis 
is  one  of  these,  and  it  has  become  a general  requirement  for  the 
management  of  military  resources. 

A.  REQUIREMENT  FOR  COST-EFFECTIVENESS  ASSESSMENT 

Department  of  Defense  Instruction  (D0D1)  70*41.3,  Economic 
Analysis  and  Program  Evaluation  for  Resource  Management  (1972) 
establishes  the  general  policy  for  cost-effectiveness  analyses 


25 


and  outlines  the  requirements  to  which  such  analyses  must  adhere. 
Cost-effectiveness  analyses  are  required  for  first-time  funding 
of  projects  and  periodically  for  on-going  activities.  This  policy 
has  been  promulgated  in  each  of  the  Services  by  implementing 
instructions  and  has  been  interpreted  in  numerous  writings. 

There  are  two  ways  of  evaluating  the  cost-effectiveness  of 
a'.ternative  military  systems.  Given  two  systems  of  the  same 
cost,  one  would  prefer  the  system  that  provides  greater  effective- 
ness. Given  two  systems  of  the  same  level  of  effectiveness,  one 
would  prefer  the  system  that  costs  less.  All  studies  of  computer- 
assisted  and  computer-managed  instruction  have  used  the  second 
approach.  Computer-based  instructional  systems  have  been  designed 
to  provide  the  same  degree  of  effectiveness  (student  achievement) 
as  the  method  of  Instruction  they  might  replace  (conventional 
instruction).  Therefore,  these  alternative  methods  of  instruc- 
tion must  be  evaluated  in  terms  of  differences  in  their  costs. 

To  date,  evaluations  of  computer-assisted  and  computer- 
managed  instruction  have  addressed  questions  concerning  the  tech- 
nical and  operational  feasibility  of  these  methods  of  instruction, 
including  the  design  of  courses  for  these  methods  of  instruction. 
Most  studies  have  addressed  the  effectiveness  of  instruction; 
some  treated  costs  and  some  treated  cost-effectiveness.  However, 
cost  and  cost-effectiveness  appear  to  have  been  secondary  con- 
siderations in  these  studies.  Table  1 lists  30  studies  that  gen- 
erated 48  data  sets  on  the  effectiveness  of  CAI  or  CMI  in  military 
training;  only  eight  of  these  provided  any  data  on  the  costs  of 
these  programs,  most  of  which  were  experimental  rather  than  opera- 
tional in  nature;  only  five  evaluated  cost-effectiveness. 

At  some  point,  the  cost-effectiveness  of  computer-based 
instruction  will  have  to  be  established  in  a definitive  fashion. 
Taken  as  a group,  these  30  studies  do  not  provide  a sufficient 
basis  on  which  to  evaluate  the  cost-eflectiveness  of  either 
computer-assisted  or  computer-managed  instruction. 
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TABLE  1 . OATA  ON  EFFECTIVENESS  AND  COSTS  IN  EVALUATIONS  OF 
COMPUTER-ASSISTEO  AND  COMPUTER-MANAGED  INSTRUCTION 


) 

i 

4 
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There  is  an  extensive  literature  that  describes  procedures 
for  conducting  cost,  cost-effectiveness,  and  cost-benefit  analy- 
ses, e.g.,  Sassone  and  Schaffer  (1978),  Quade  and  Boucher  (1968), 
and  Fisher  (1971).  Applications  of  these  procedures  to  a variety 
of  fields  may  be  found  in  Alfandary-Alexander  (1968)  and  Goldman 
(1967).  The  application  of  these  procedures  to  military  train- 
ing is  described  by  Doughty,  Stern,  and  Thompson  (1976)  and 
Swope  (1976).  Resource  estimation  procedures  associated  with 
the  conduct  of  military  training  are  identified,  among  others, 
by  Hess  and  Kantar  (1977)  and  Braby,  Henry,  Parrish,  and  Swope 
(1976) . 
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B.  CRITIQUE  OF  THE  LITERATURE 


Computer-assisted  and  computer-managed  instruction  in  mill-  ] 

tary  training  have  been  evaluated  in  about  30  studies  conducted  j 

since  1968.  Most  of  these  were  experiments  conducted  with  R&D 
funds,  while  a few  approximated  operational  conditions.  Most  of 
these  studies  were  concerned  primarily  with  the  effectiveness  of 
computer-based  instruction,  a few  with  its  costs;  some  implied 
that  their  results  related  to  cost-effectiveness  but  did  not 
actually  perform  any  analyses.  The  critique  that  follows  discus- 
ses the  following  issues:  I 


• The  scope  of  the  studies  j 

• The  measures  of  effectiveness  used  ] 

• The  incompleteness  of  cost  information  ] 

• Treatment  of  expenditures  as  costs  j 

• The  incomplete  range  of  alternatives  considered.  J 

1 . Scope  of  the  Studies  ] 


Most  of  the  data  were  collected  under  programs  funded  through 
the  RDT&E  appropriation,  i.e.,  Exploratory  Development  (6.2)  and 
Advanced  Development  (6.3).  Such  programs  are  generally  small 
in  scale  with  regard  to  numbers  of  students,  hours  of  instruc- 
tion, and  duration.  The  48  data  sets  developed  in  these  programs 
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are  based  on  totals  of  about  800  hours  of  computer-based  Instruc- 
tional materials,  9,000  students,  and  400,000  student  hours. 

The  four  data  sets  for  the  Advanced  Instructional  System  (AIS) , 
a demonstration  rather  than  an  experimental  program,  account  for 
approximately  40  percent  of  all  instructional  hours,  over  70  per- 
cent of  the  students,  and  over  85  percent  of  the  student  hours  in 
these  30  studies.  Of  the  remaining  44  programs,  only  21  involve 
more  than  10  hours  of  Instruction  and  only  18  include  more  than 
50  students  as  subjects  (see  Table  2). 

TABLE  2.  DISTRIBUTIONS  OF  COURSE  LENGTH  AND  NUMBER  OF 
STUDENTS  IN  48  EVALUATIONS  OF  MILITARY  CAI 
AND  CM  I INSTRUCTION 


The  results  obtained  with  respect  to  student  achievement  and 
time  required  to  complete  courses  in  short-term  experiments  may 
differ  from  those  found  in  large-scale,  long-term  operational 
training  programs.  Some  data  in  Chapter  III  suggest  that  similar 
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results  are  found  In  both  cases.  However,  the  extrapolation  of 
cost  data  from  experiments  to  operational  programs  is  partic- 
ularly inappropriate.  Management  and  accounting  of  resources 
differ  between  operational  programs  and  those  funded  through 
Research,  Development,  Test  and  Evaluation.  In  addition,  dif- 
ferences in  cost  may  be  anticipated  between  training  in  opera-  • 
tional  and  in  research  settings  because  of  different  scales  of 
operation,  different  organization  of  the  training  program,  and 
differences  in  the  utilization  of  equipment  and  personnel. 

2.  Measures  of  Effectiveness  Used 

The  purpose  of  military  training  is  to  provide  personnel 
with  the  skills  and  knowledge  required  to  perform  specific  tasks 
in  the  operational  forces  under  both  peacetime  and  combat  situa- 
tions. Thus,  the  effectiveness  of  alternative  methods  of  in- 
struction must  be  evaluated  by  comparing  how  well  personnel, 
trained  by  either  method,  perform  in  operational  units.  Meas- 
urement of  performance  of  graduates  on  jobs  in  the  field  implies 
a system  for  postgraduate  monitoring  of  students  for  some  period 
of  time  after  their  assignment  to  duty  stations.  This  measure 
was  not  used  in  any  of  the  studies.  At  present,  data  on  the 
effectiveness  of  training  in  schools  are  not  collected  system- 
atically; the  data  that  are  collected  consist  of  supervisors' 
opinions  about  the  Job  performance  of  graduates.  Such  data  are 
subjective  in  nature  and  may  be  influenced  by  factors  not  related 
to  training,  e.g.  relevance  of  the  training  course  to  the  actual 
Job,  nature  of  the  work  environment,  personality,  and  so  on. 

Instead,  we  found  that  the  following  measures  of  effective- 
ness were  used  in  these  studies:  (1)  student  achievement  on 
tests  administered  during  and/or  at  the  end  of  course,  (2)  the 
length  of  time  required  for  students  to  complete  a course, 

(3)  academic  attrition  rates,  and  (4)  student  and  instructor  at- 
titudes. None  are  appropriate  measures  of  effectiveness.  The 
use  of  multiple  measures  may  lead  to  contradictory  conclusions, 
unless  they  can  be  combined  in  a meaningful  way. 
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Training  course  curricula  are  based  on  assessment  of  Bkills 
required  of  personnel  in  operational  billets  as  determined,  for 
example,  by  a task  analysis.  To  the  extent  that  such  assessments 
are  valid,  student  achievement  in  school  as  shown  by  test  results 
may  serve  as  a proxy  (or  predictor)  of  future  field  performance 
and,  hence,  training  effectiveness.  However,  correlations  between 
performance  at  school  and  on  the  .lob  have  not  been  established  for 
any  method  of  instruction,  and  the  use  of  results  collected  only 
at  schools  cannot  be  taken  to  be  conclusive.  On  the  whole,  the 
data  suggest  that  student  achievement  at  school  is  about  the  same 
with  all  methods  of  Instruction  considered  in  this  paper.  The 
differences  that  were  found  are  not  thought  to  have  practical 
importance.  This  will  be  discussed  later  in  this  paper. 

To  the  extent  that  student  time  is  relevant  to  the  analysis 
of  cost-effectiveness,  it  is  a measure  of  the  cost  and  not  of  the 
effectiveness  of  training.  The  studies  would  have  gone  a long 
way  toward  fixed-effectiveness  or  net  cost  evaluations  if  the 
observed  decreases  in  the  time  needed  by  students  to  complete 
courses  given  by  computer-based  instruction  had  been  converted 
to  decreased  (or  avoided)  cost  of  instruction,  through  standard 
factors  for  pay  and  allowances  and  other  personnel-related  re- 
sources; these  decreases  would  have  to  be  offset  against  the 
costs  of  other  resources  that  are  incremental  to  the  use  of 
computer-based  instruction,  e.g.,  computers  and  courseware.  Cost 
is  similarly  associated  with  student  attrition.  With  lower  at- 
trition rates,  fewer  students  are  required  to  enter  training  to 
produce  a specified  number  of  graduates  and,  thus,  a smaller 
total  number  of  student  days  are  spent  in  training. 

Student  and  instructor  attitudes  are  qualitative  and  not  yet 
quantifiable  factors.  To  the  extent  that  student  attitudes  might 
impact  on  school  achievement,  it  might  be  manifest  in  test  per- 
formance, course  time,  and  attrition;  these  possibilities  are 
not  known  and  have  not  been  explored.  While  it  may  be  granted 
that  attitudes  might  affect  either  cost  or  effectiveness  through 
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suoh  considerations  as  later  field  performance  and  reenlistment 
rates  (including  those  of  instructors),  the  relationships,  if 
any,  are  oblique  and  remain  unknown. 

3.  Incompleteness  of  Cost  Information 

In  the  eight  studies  that  addressed  cost,  the  cost  data 
described  only  some  of  the  direct  expenditures  that  were  incur- 
red during  the  course  of  the  studies.  Some  of  this  information 
appears  to  have  been  reconstructed  after  the  fact  rather  than 
recorded  during  the  period  of  experimentation  and  it  may  not  be 
accurate . 

Training  is  an  intricate  process  that  requires  different 
types  of  resources  to  perform  a variety  of  functions.  Some  re- 
sources may  be  uniquely  associated  with  a single  method  of  in- 
struction while  others  will  be  common  to  several  methods.  For 
example,  computer  hardware  (a  type  of  resource)  is  a unique 
requirement  of  computer-based  instruction.  Development  of 
courseware  (a  function  provided  by  resources)  is  associated  with 
all  methods  of  instruction,  but  its  cost  per  unit  ( e.g .,  man- 
hours per  hour  of  instruction  developed)  appears  to  vary  widely 
between  different  methods  of  instruction.  Similarly,  instruc- 
tional personnel  are  employed  by  all  methods  of  instruction; 
while  its  nominal  cost  (per  hour  of  instruction)  may  be  constant 
between  instructional  methods,  its  effective  cost  (per  student 
hour)  depends  on  the  student : instructor  ratio  characteristic  of 
each  method  of  instruction.  In  comparing  the  costs  of  alterna- 
tive methods  of  instruction,  It  is  necessary  to  account  for  all 
resources  whose  costs  may  differ  between  alternatives.  That  is, 
all  such  costs  must  be  considered  relevant  to  the  analysis. 

It  is  a relatively  straightforward  exercise  to  identify 
the  resources  for  which  data,  are  needed  to  compare  the  costs  of 
computer-based  instruction  and  of  other  methods  of  instruction. 
Table  3 displays  a list  of  these  resources,  at  a major  category 
level,  developed  from  our  reading  of  the  literature.  None  of 
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TABLE  3.  RESOURCES  REQUIRED  TO  SUPPORT 
VARIOUS  METHODS  OF  INSTRUCTION 
(MAJOR  CATEGORIES  ONLY) 


RESOURCE  (TYPE  OR  FUNCTION) 
PROGRAM  DEVELOPMENT 
PR06RAM  DESIGN 
INSTRUCTIONAL  MATERIALS 
CONVENTIONAL 
INDIVIDUALIZED  INSTRUCTION 
PROGRAMMING 
RflST  UNIT  PRODUCTION 
COMPUTER-BASED 
PROGRAMMING 

CODING 

PROGRAM  DELIVERY 
INSTRUCTION 
INSTRUCTORS 

INSTRUCTIONAL  SUPPORT  PERSONNEL 
EQUIPMENT  ANO  SERVICES 

LABORATORY  (INCLUDING  SIMULATORS) 
MEDIA  DEVICES 
COMPUTER  8YSTEMS 
COMMUNICATIONS 

MATERIALS  (INCLUDING  CONSUMABLES) 

FACILITIES 

PROGRAM  MANAGEMENT  AND  ADMINISTRATION 
8TUDENT  PERSONNEL 
PAY  AND  ALLOWANCES 
OTHER  (TEMPORARY  DUTY, 

PERMANENT  CHANGE  OF  STATION,  ETC.) 


H»-7#-7  H»-7» 
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the  eight  studies  which  reported  costs  of  computer-based  instruc- 
tion provided  data  for  all  items  on  this  list. 

The  extent  of  incompleteness  may  be  judged  by  the  following 
comments.  Computer  hardware  (either  leased  or  procured)  and 
courseware  development  are  two  categories  of  major  impact  assoc- 
iated with  the  cost  of  computer-based  instruction;  three  studies1 
provided  no  information  regarding  courseware  costs  and  two*  pro- 
vided no  information  regarding  computer  hardware  costs.  Only 
four  of  the  studies  provided  cost  information  for  anything  other 
than  computer  hardware  or  courseware  development* ; in  two  of  these 
cases,  the  only  other  costs  reported  were  for  compressed  air  and 
carrels  for  PLATO  IV  terminals,  relatively  minor  items. * 

More  notable  than  the  incompleteness  of  data  on  the  costs 
of  computer-based  instruction  is  the  lack  of  information  regard- 
ing the  costs  of  alternative  methods  of  instruction.  All  studies 
compared  the  effectiveness  of  computer-based  and  an  alternative 
method  of  instruction  (generally  conventional  instruction),  using 
the  measures  described  above.  Only  one  of  the  eight  studies  ad- 
dressing costs  (Crawford,  Hurlock,  et  al.,  1976)  compared  the  cost 
of  the  experimental  program  to  that  of  the  method  by  which  the 
same  material  was  normally  taught.  Two  studies  provided  in- 
complete information  on  courseware  development  for  individualized 
Instruction  (U.S.  Army  Ordnance  Center  and  School,  1975;  Dallman, 
DeLeo  et  al. , 1977);  the  data  were  taken  from  other  studies.  In 
essence,  even  when  some  data  were  provided  on  the  costs  of 


‘Ford,  Slough,  and  Hurlock  (1972);  Crawford,  Hurlock,  et  al. 
(1976);  Steinkerchner , Deignan,  et  al.  (1977). 

2Carson,  Graham,  et  al.  (1975);  Keesler  AFB  (197*0. 

’Crawford,  Hurlock,  et  al.  (1976);  Carson,  Graham,  et  al.  (1975); 
Steinkerchner,  Deignan,  et  al.  (1977);  Dallman,  DeLeo,  et  al . 
(1977). 

"Steinkerchner,  Deignan,  et  al.  (1977);  Dallman,  DeLeo,  et  al. 
(1977). 
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computer-based  instruction,  comparable  cost  data  were  not  pro- 
vided for  alternative  methods  of  instruction. 

4.  Expenditures  as  Cost 

Most  studies  that  considered  the  cost  of  training  treated 
expenditures  of  funds  during  the  course  of  the  experimental  pro- 
grams as  equivalent  to  the  cost  of  training.  For  example,  in  ex- 
periments using  PLATO  IV,  no  distinction  was  made  between  funds 
expended  for  the  purchase  of  terminals  and  for  access  to  the 
central  processor.  The  expenditure  for  terminals  is  an  invest- 
ment in  long-lived  assets  that  can  provide  training  both  during 
and  after  the  period  of  the  experiment;  thus,  oply  a fraction  of 
the  procurement  cost  is  a cost  of  training,  t.e.,  during  the 
limited  time  of  the  experiment.  On  the  other  hand,  expenditure 
for  purchase  of  central  processor  time  is  strictly  a cost  of  the 
experimental  program;  access  to  the  central  processor  during  the 
experiment  provided  no  residual  capability  to  support  training 
after  the  experiment  was  completed.  The  simple  sum  of  expendi- 
tures for  terminals  (investment)  and  access  to  the  computer 
(operations)  is  meaningless  for  any  period  less  than  an  assumed 
total  life-cycle  of  the  system.  Two  studies  (Hurlock  and  Slough, 
1976  and  Crawford,  Hurlock,  et  al . , 1976)  were  exceptions  to 
this  type  of  treatment.  In  these  cases,  a portion  of  the  re- 
corded expenditures  was  translated  into  estimated  costs  of  an 
operational  PLATO  IV  training  program. 

Translations  from  current  expenditures  to  costs  that  can 
be  summed  into  meaningful  totals  require  resort  to  some  form  of 
analytic  framework  or  model,  and  a formal  model  is  called  for 
in  cost-effectiveness  analysis.  It  imposes  the  discipline  of 
explicitly  identifying  all  inputs,  assumptions,  and  relation- 
ships so  that  alternatives  can  be  compared  in  a consistent  manner. 
For  example,  alternative  methods  of  instruction  may  be  affected 
differently  by  such  conditions  as  limits  on  utilization  of  equip- 
ments (e.g.  , attainable  terminal  hours),  availability  of  required 
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resources,  (e.g.,  media  devices)  and  differences  in  lifespans 
of  various  resources  (e.g.  , computer  hardware  versus  laboratory 
equipment);  yet,  such  conditions  are  difficult  to  treat  completely 
and  consistently.  Models  which  would  call  attention  to  the  full 
range  of  inputs  and  assumptions  and  the  ways  they  are  incorporated 
into  evaluations  of  cost  were  missing  from  the  studies  reviewed. 

5.  Incomplete  Range  of  Alternatives  Considered 

All  studies  compared  student  achievement  and  the  times 
needed  by  students  to  complete  the  same  course  given  by  conven- 
tional instruction  and  by  computer-based  instruction.  A course 
given  by  conventional  instruction  must  be  rearranged  into  a series 
of  lessons  and  tests  in  order  to  be  given  by  computer-assisted  or 
computer-managed  instruction,  or  by  individualized  instruction 
without  computer  support.  During  the  process  of  revision,  course 
materials  are  reviewed  and  modified;  if  a task  analysis  is  per- 
formed, material  that  is  no  longer  relevant  will  be  dropped  and 
new  material  may  be  added.  The  result  is  that  the  course  materials 
used  with  a new  method  of  Instruction  are  rarely  identical  to 
those  used  in  the  old  course;  note  that  the  revised  course 
materials  could  be  taught  by  any  method  of  instruction,  including 
conventional  instruction.  Figure  1 describes  the  steps  involved 
in  changing  a course  from  conventional  to  computer-based  or  any 
other  form  of  instruction:  the  course  materials  are  revised  and 
restructured  into  an  appropriate  format.  Each  step  in  this 
process  implies  an  expenditure  of  resources;  each  method  of  in- 
struction implies  a different  final  cost,  and  may  yield  different 
levels  of  instructional  effectiveness. 

Thus,  all  comparisons  of  computer-based  instruction  with 
conventio'nal  instruction  produce  results  (generally  student 
time  savings)  that  may  be  due  to  the  new  method  of  instruction 
and/or  to  the  process  of  course  revision  (which  may  also  shorten 
or  lengthen  the  course).  There  is  one  study  where  a computer- 
based  course  was  compared  indirectly  to  its  revised,  conventional 
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version;  all  other  comparisons  are  with  the  original  conventional 
course.  There  are  some  data  where  a computer-based  course  was 
compared  to  its  revised,  individualized  version. 


+•  CONVENTIONAL  INSTRUCTION 


CONVENTIONAL 
INSTRUCTION 
» ■ ■ 


INDIVIDUALIZED 

COURSE 

MATERIALS 


REVISED 

COURSE 

MATERIALS 


INDIVIDUALIZED  INSTRUCTION 


COMPUTER-ASSISTED  INSTRUCTION 


COMPUTER-MANAGED  INSTRUCTION 


BARLINE 

EXPERIMENTAL 

COURSE 

COURSE 

FIGURE  1.  Steps  Involved  In  modifying  a course  from 

conventional  to  Individualized  or  computer- 
based  Instruction. 

C.  SUMMARY 

The  data  base  used  to  evaluate  the  cost-effectiveness  of 
computer-assisted  and  computer-managed  instruction  in  military 
training  has  the  following  limitations: 

• Thirty  studies,  conducted  since  1968,  provide  48  data 
sets  on  the  effectiveness  and  8 data  sets  on  the  costs 
of  computer-based  instruction.  About  half  of  the  studies 
are  based  on  10  or  less  hours  of  Instruction;  about  half 
of  the  studies  are  based  on  50  or  less  students;  a few 
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studies  (on  computer-managed  instruction)  involve  longer 
courses  (2  to  10  months)  and  larger  numbers  of  students 
(600  to  2500). 

• The  most  relevant  measure  of  effectiveness  of  Instruction 
is  the  performance  of  graduates  of  courses  on  the  job  in 
an  operational  unit.  Such  data  are  not  now  available. 

All  studies  use  student  achievement  at  school  as  a meas- 
ure of  effectiveness.  The  relation  between  student 
achievement  at  school  and  performance  on  the  Job  has  not 
been  demonstrated.  Some  measures  of  effectiveness  that 
have  been  used  {e.g.,  student  time  saved  and  student  at- 
trition in  courses)  should  be  treated  as  measures  of 

l.  o s t « 

• The  cost  data  derived  from  these  studies  are  generally 
incomplete;  the  cost  data  reported  in  experiments  do  not 
extrapolate  readily  to  operational  settings  because  of 
major  differences  in  training  organizations  and  accounting 
procedures.  No  data  are  provided  that  permit  comparisons 
between  the  costs  of  computer-based  and  conventional  in- 
struction. 

• None  of  the  studies  provide  an  explicit  distribution  of 
costs  over  some  specified  life  cycle  for  comparable 
methods  of  instruction;  none  provide  a model  for  use  in 
estimating  costs. 

• Most  comparisons  of  student  achievement  with  computer- 
based  and  conventional  instruction  provide  confounded 
results  that  may  be  attributed  either  to  the  method 

of  instruction  and/or  to  revision  of  course  materials. 
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III.  EFFECTIVENESS  OF  COMPUTER-BASED  INSTRUCTION 


Military  training  is  intended  to  provide  the  skills  and 
knowledge  required  to  perform  various  tasks  in  operational  units. 
Thus,  the  effectiveness  of  computer-based  instruction  for  teach- 
ing a particular  course  should  be  compared  to  that  of  conven- 
tional instruction  by  measuring  how  well  graduates  taught  either 
way  perform  the  same  tasks  in  the  field.  Such  data  were  not 
found  in  the  research  literature  dealing  with  the  effectiveness 
of  computer-based  and  conventional  instruction  in  military  train- 
ing. 

Instead,  we  found  that  the  following  measures  of  effective- 
ness have  been  used: 

• The  amount  and/or  quality  of  information  and  skills 
acquired  by  students  at  school  (end-of-course  achieve- 
ment ) 

• The  amount  of  time  required  by  students  to  complete  a 
course  (student  time  savings) 

• The  number  of  students  who  do  not  complete  a course  for 
academic  reasons  (academic  attrition) 

• Attitudes  of  students  (acceptability  of  computer-based 
Instruction  to  students) 

• Attitudes  of  instructors  (acceptability  of  computer- 
based  instruction  to  instructors). 

These  measures  can  be  collected  conveniently  at  schools  or 
experimental  sites  before  students  scatter  tc  other  assignments 
but  they  are  not  necessarily  appropriate  for  evaluating  the  ef- 
fectiveness of  computer-based  instruction.  Various  limitations 
of  these  measures  were  discussed  in  Chapter  II. 
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Here  the  effectiveness  of  CAI  and  CMI  is  considered  on  the 
basis  of  the  evidence  provided  by  military  research  studies  and 
with  explicit  recognition  of  some  major  limitations  to  this  evi- 
dence, as  follows:  (1)  measures  of  student  achievement  at 
school  must  be  validated  by  data  on  performance  on  the  Job  and 
(2)  measures  of  time  saved  by  students  at  school  and  measures 
of  academic  attrition  at  school  should  be  treated  as  measures 
of  cost  rather  than  of  effectiveness. 

In  general,  the  military  interest  in  CAI  and  CMI  Is  based 
on  the  premise  that  these  methods. of  instruction  may  save  stu- 
dent training  time  with  little,  if  any,  loss  in  student  achieve- 
ment. The  Interest  of  schools  and  colleges  is  based  on  the  pre- 
mise that  CAI  may  provide  the  same  or  greater  student  achievement 
than  that  provided  by  conventional  Instruction;  there  is  much 
less  concern  here  for  the  amounts  of  time  spent  by  students 
under  various  methods  of  instruction.  Schools  and  colleges 
have  shown  little  interest  in  CMI.  A brief  summary  of  the  find- 
ings on  the  effectiveness  of  CAI  for  instruction  in  schools  and 
colleges,  based  primarily  on  evaluations  of  PLATO  IV  and  TICCIT 
in  community  colleges,  appears  in  Appendix  C. 

A.  SOURCES  OF  INFORMATION 

CAI  and  CMI  instruction  have  been  evaluated  in  about  30 
studies  conducted  by  the  military  Services  over  the  period  of 
1968  to  1978.  These  studies  are  summarized  in  a series  of 
tables  in  Appendix  D.  These  studies  sample  a wide  variety  of 
courses  in  technical  training,  e.g.,  basic  electronics,  elec- 
tricity, vehicle  repair,  inventory  management,  fire  control, 
and  precision  equipment,  among  others  (see  Table  *0 . The  courses 
Include  cognitive  skills  (knowledge,  theory,  and  rules)  and  per- 
formance-oriented skills  (hands-on  maintenance,  checkout,  and 
repair)  at  a wide  range  o!1  skill  levels.  There  is  no  overlap 
between  the  courses  used  in  evaluations  of  CAI  and  CMI. 
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TABLE  4.  COURSES  USED  IN  VARIOUS  STUDIES  OF  CAI  AND  CMI 


Courses 

Basic  electronics 

Electricity 

Machinist 

Training  materials  development 
Recipe  conversion 
Aircraft  panel  operation 
Medical  assistant 
Vehicle  repair 
Weather 

Tactical  coordinator  (S-3A) 

Fire  control  technician 
Aviation  familiarization 
Aviation  mechanical  fundamentals 
Inventory  management 
Materiel  facilities 
Precision  measuring  equipment 
Weapons  mechanic 


B.  STUOENT  ACHIEVEMENT 


The  effectiveness  of  CAI  and  CMI  as  methods  of  instruction, 
compared  to  conventional  or  individualized  instruction,  has 
been  measured  only  by  performance  of  students  on  tests  admin- 
istered at  schools,  rather  than  by  performance  on  the  Job  after 
graduation.  Student  achievement  at  school  might  predict 
quality  of  performance  on  the  Job  but  correlations  between 
these  two  measures  have  not  been  established  for  conventional 
or  for  computer-based  instruction.  The  Services  evaluate  some 
courses  by  means  of  supervisors1  ratings  of  the  performance  of 
graduates  on  the  Job;  however,  these  results  are  qualitative 
in  nature  and  have  not  been  collected  systematically.  Data  on 
student  achievement  at  school,  found  in  various  studies,  are 
summarized  in  Table  5. 

In  40  comparisons,  student  achievement  with  CAI  was  about 
the  same  as  with  conventional  Instruction  in  24  cases,  superior 
in  15,  and  inferior  in  one.  The  differences  in  performance, 
although  statistically  significant,  were  Judged  not  to  have 
practical  significance.  In  eight  comparisons  of  CMI  with  con- 
ventional instruction,  no  significant  differences  were  found 
in  student  achievement  at  school. 

In  addition  to  these  results,  there  were  five  cases  where 
student  achievement  on  CAI  was  compared  to  that  on  individual- 
ized instruction.  Achievement  was  the  same  in  four  cases, 
and  superior  with  CAI  in  one. 

The  fact  that  student  achievement  with  CAI  and  CMI  is 
about  the  same  as  that  with  conventional  instruction  or  individ- 
ualized instruction  is  also  a direct  consequence  of  the  fact 
that  students  instructed  by  CAI  and  CMI  are  held  in  these 
courses  until  they  master  all  lessons.  The  critical  variable 
thus  becomes  the  amount  of  time  needed  to  complete  courses 
given  by  computer-based  instruction. 


42 


STUDENT  ACHIEVEMENT  AT  SCHOOL  FOR  CAI  AND  CMI , COMPARED 
TO  CONVENTIONAL  INSTRUCTION,  IN  MILITARY  TRAINING 


Best  Available  Copy 


C.  STUDENT  TIME  SAVINGS 


Most  studies  take  the  amount  of  student  time  saved  to  com- 
plete courses  given  by  CAI  or  CMI,  in  comparison  to  convention- 
al instruction,  as  a measure  of  effectiveness.  As  pointed  out 
above,  the  amount  of  student  time  saved  is  a measure  of  cost. 
Student  time  savings  reported  in  30  studies  are  shown  in  Table  6 
and  summarized  in  Table  7. 

When  the  findings  for  CAI  and  CMI  are  combined,  computer- 
based  instruction  appears  to  save  about  one-third  of  the  time 
required  by  students  to  complete  the  same  courses  when  given  by 
conventional  instruction.  However,  there  is  a wide  variation  in 
the  amounts  of  savings  that  have  been  reported.  The  amounts  of 
student  time  saved  by  CAI  and  CMI  cannot  be  compared  because  in 
no  case  was  the  same  course  given  by  both  methods  of  instruction. 
Two  major  uncontrolled  variables  in  these  studies  are  the  un- 
known quality  of  the  instructional  materials  used  in  the  various 
comparisons  and  uncertainty  that  the  same  amounts  of  course 
materials  were  used  in  both  methods  of  instruction.  This  argues 
against  trying  to  interpret  apparent  differences  in  the  amounts 
of  student  time  saved  by  CAI  or  CMI,  or  by  different  courses, 
and  so  on. 

There  are  three  Instances  where  the  use  of  CAI  increased 
rather  than  decreased  student  training  time  and  one  where  its 
effect  was  zero.  These  may  be  attributed  to  inadequate  prepara- 
tion of  course  materials  or  other  factors  not  explained  in  these 
experiments.  These  atypical  results  occur  only  in  some  initial 
studies  and  not  in  more  recent  ones;  in  any  case,  such  findings 
would  not  be  recommended  for  operational  use. 

The  fact  that  CAI  and  CMI  save  student  training  time  is 
consistent  with  well-known  Information  about  wide  differences 
in  student  ability  (as  represented  in  the  normal  distribution 
curve)  and  in  the  amounts  of  relevant  knowledge  held  by  students 
at  the  start  of  any  course.  In  conventional  instruction  with  a 
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TABLE  6.  AMOUNT  OF  STUDENT  TIME  SAVED  IN  COURSES  GIVEN  BY  CAI  AND  CMI , 
COMPARED  TO  CONVENTIONAL  INSTRUCTION,  IN  MILITARY  TRAINING 


TABLE  7.  AMOUNTS  OF  STUOENT  TRAINING  TIME  SAVED 
BY  CAI  ANO  CMI , COMPARED  TO  CONVENTIONAL  INSTRUCTION 


Method  of 
Instruction 

Number  of 
Comparisons 

CAI 

40 

Student  time  savings, 
compared  to  conventional 
instruction,  percent 

Median 

Range 

20 

•31  to  89 

44 

12  to  69 

32 

•31  to  89 

12-25-7H0 


fixed  amount  of  time,  these  differences  lead  to  variations  in 
the  amounts  of  knowledge  acquired  by  the  end  of  the  course, 
i.e.,  as  shown  by  a distribution  of  final  grades.  In  individual- 
ized instruction,  whether  computer-based  or  not,  each  student 
proceeds  at  his  own  pace  and  differences  between  students  in- 
fluence the  amounts  of  time  they  need  to  complete  the  course 
more  than  it  does  the  amounts  of  information  acquired.  Most 
of  the  time  savings  in  Individualized  instruction  are  produced 
by  those  students  for  whom  the  rate  of  progress  set  in  conven- 
tional instruction  would  be  too  slow;  typically  that  rate  might 
be  one  that  permits  about  90  percent  of  the  students  to  complete 
the  course  during  the  fixed  period  of  time. 

Almost  all  of  the  data  shown  in  Table  6 represent  time 
savings  found  in  experiments  or  operational  tests  over  short 
time  periods  and  with  limited  numbers  of  students.  Figure  2 
shows  the  amounts  of  time  required  by  about  11,000  students  to 
complete  four  courses  on  the  Air  Force  Advanced  Instructional 
System  (AIS),  Lowry  AFB  over  24  months  ending  September  1978. 


■ PRECISION  MEASURM6  EOUTMENT 
= WEAPON  MECHAMC 
= MVENTORY  MANAGEMENT 
= MATEMEL  FAOUTKS 
= NEW  SELF-PACED  BLOCKS  MTNOOUCEO 
= SOME  SELF-PACED  BLOCKS  REVISED 
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FIGURE  2.  Days  Required  to  Complete  Four  Courses 
on  Air  Force  Advanced  Instructional 
System,  Lowry  AFB,  October  1976- 
September  1978 


It  is  clear  that  the  initial  savings,  such  as  might  be  reported 
J.i  an  experiment,  are  maintained  over  time  and,  despite  monthly 
fluctuations,  tend  to  increase.  The  reasons  for  these  reduc- 
tions and  fluctuations  have  not  been  explored;  they  could  be 
due,  at  least  in  part,  to  periodic  revisions  in  the  courses 
(indicated  on  the  figure),  to  improved  control  over  the  new 
method  of  instruction,  to  variations  in  student  aptitude  and 
to  turnover  among  instructors.  Similar  reductions  in  student 
time  are  shown  in  Pig.  3 for  Pbc”.t  12,000  graduates  in  three 
courses  on  the  Navy  Computer  Managed  Instruction  System  at 
Naval  Air  Technical  Training  Center.,  Millington,  Tennessee, 
over  a 15-month  period  ending  May  1978.  No  significant  changes 
were  made  in  these  courses  during  this  period. 

D.  STUDENT  ATTRITION 

Since  the  method  of  instruction  may  influence  the  number 
of  students  who  can  successfully  complete  a course,  the  rate  of 
academic  attrition  associated  with  alternative  methods  of  in- 
struction is  a matter  of  concern.  As  noted  previously,  the  rate 
of  attrition  is  a measure  of  the  cost  of  instruction  since  it 
Influences  the  number  of  students  needed  to  enter  a course  in 
order  to  produce  a specified  number  of  graduates.  Attrition 
for  nonacademic  reasons,  such  as  for  medical  or  disciplinary 
reasons.  Is  not  considered  here.  It  should  also  be  recognized 
that  the  rate  of  attrition  observed  in  a course  may  be  influenced 
from  time  to  time  by  policy  decisions  on  standards  for  recruit- 
ment and  the  number  of  graduates  to  be  produced  by  various 
courses.  Such  influences,  if  present,  are  not  addressed  here. 

Meaningful  data  on  student  attrition  related  to  computer- 
based  instruction  should  come  from  steady-state  applications 
and  not  from  short-term  experiments.  This  condition  is  met 
marginally  by  the  Air  Force  Advanced  Instructional  System  (AIS), 
where  four  courses  were  increasingly  implemented  on  a computer- 
managed  Instructional  system  over  the  period  of  197**  to  1978  and 
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FIGURE  3.  Time  (Student  Contact  Hours)  Required  to  Complete 
Three  Courses  In  Propulsion  Engineering  Basics  on 
the  Navy  Computer  Managed  Instruction  System, 
Millington,  Tennessee,  March  1977  to  May  1978 
(Students  at  Navy  Training  Center,  Great  Lakes, 
Illinois) 


by  the  Navy  Computer  Managed  Instruction  System,  where  data  are 
available  on  seven  courses  before  and  after  implementation  in 
March  1977- 

Figure  4 shows  that,  compared  to  previous  rates,  academic 
attrition  may  have  increased  in  the  four  courses  implemented  on 
AIS.  Note,  however,  that  academic  attrition  appeared  to  rise 
in  all  (non-AIS)  courses  at  Lowry  AFB  over  the  same  period; 
thus,  it  is  not  obvious  that  the  increased  attrition  in  the  AIS 
courses  should  be  attributed  primarily  to  the  introduction  of 
CMI  instruction. 

Figure  5 shows  academic  attrition  for  seven  courses  be- 
fore and  after  implementation  on  the  Navy  Computer  Managed  In- 
struction System.  The  average  rate  of  academic  attrition  in 
these  courses  was  3.2  percent  before  and  4.6  percent  after  im- 
plementation on  the  Navy  CMI  system  (it  Increased  in  six 
courses  and  decreased  in  one).  Data  on  comparable  courses  not 
on  CMI  during  the  same  period  were  not  provided. 

Little  data  are  available  on  academic  attrition  during 
experiments.  Longo  (1972)  says  that  academic  attrition  was 
about  the  same  for  two  courses  in  basic  electronics  taught  by 
CAI  or  by  conventional  instruction;  Giuntl  and  Longo  (1971b) 
say  that  attrition  was  22  percent  lower  for  the  CAI  group  in 
another  study;  there  were  few  students  (66  - 186)  in  any  of  the 
studies  summarized  here.  The  use  of  CAI  on  four  Special  Pur- 
pose Vehicle  Repairman  courses  at  Chanute  AFB  produced  no 
significant  effect  on  academic  attrition  over  a 9-month  period 
(Dallman,  DeLeo,  Main,  and  Gillman,  1977);  about  300  students 
were  involved.  Initial  results  for  four  courses  on  the  Navy 
CMI  system  In  1975  suggested  that  there  were  no  effects  on 
student  attrition  at  that  time  (Carson,  Graham,  Harding,  et  al., 
1975). 
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Academic  Attrition  In  Seven  Courses  Before  and 
After  Implementation  on  Navy  Computer  Managed 
Instruction  System 


In  summary,  only  two  CMI  systems,  the  Air  Force  AIS  and 
Navy  CMI,  have  received  extended,  though  still  limited,  use 
in  military  training.  Academic  attrition  may  have  increased  in 
courses  taught  this  way,  compared  to  attrition  with  conventional 
instruction  during  prior  periods.  Since  these  comparisons  do 
not  take  into  account  possible  changes  in  the  qualifications  of 
students  over  the  sane  time  periods,  the  available  data  sug- 
gest but  do  not  prove  that  CMI  may  increase  academic  attrition 
over  that  found  with  conventional  instruction. 

E.  ATTITUDES  OF  STUDENTS  AND  INSTRUCTORS 

Attitudes  of  students  and  instructors  to  CAI  or  CMI  in 
military  training,  compared  to  conventional  instruction,  are 
noted  here  only  as  lalitative  aspects  of  these  methods  of  in- 
struction. Most  of  the  data  came  from  experiments  of  short 
duration.  Data  on  student  attitudes  towards  CAI  or  CMI  are  found 
in  39  of  the  40  reports  summarized  in  Appendix  D.  As  shown  in 
Table  8,  students  almost  always  favor  CAI  or  CMI  over  conven- 
tional instruction,  or  at  least  say  so  when  asked;  they  are  un- 
favorable to  CAI  in  one  case  and  find  no  difference  in  another. 

TABLE  8.  ATTITUDES  OF  STUDENTS  AND  INSTRUCTORS  COMPARING.  . 

CAI  OR  CMI  TO  CONVENTIONAL  INSTRUCTION  IN  MILITARY  TRAINING'3) 


Attitude  to  CAI/CM 

Student* 

Instructor*  i 

CAI 

CM 

CAI 

CM 

Favorable 

29 

S 

1 

• 

No  dlftoranca 

1 

- 

• 

• 

Unfavorable 

1 

• 

4 

4<» 

No  report 

1 

- 

27 

4 

Total 

32 

s 

32 

» 

*****  Mi  m Mmtor  W nfrt»  tvMurtud  I*  A*pM*x  D. 

*k*FmnMi  l«  CM  it  irrt,  duuftof  (•  wfatwaMi  *ot  *4  itufy.  > 
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Instructors'  attitudes  are  reported  only  in  9 of  these  40 
comparisons;  instructors  are  unfavorable  to  CAI  or  CMI  in  8 of 
these  9 cases  and  favorable  to  CAI  only  in  1. 

Instructors  of  courses  taught  by  CAI  or  CMI  have  not  re- 
ceived much  attention  by  researchers.  According  to  two  studies 
still  in  draft  (February  1979)*  only  half  of  54  instructors 
sampled  in  1977  at  the  Naval  Air  Technical  Training  Center, 
Millington,  Tennessee,  believe  that  Individualized  instruction 
is  as  effective  as  conventional  instruction.*  The  training  of 
Instructors  is  still  oriented  largely  towards  conventional  in- 
struction, and  instructors  assigned  to  CMI  receive  little  guid- 
ance on  how  to  conduct  such  courses. 

F.  COMPARISON  OF  TIME  SAVINGS  FOUND  WITH  INDIVIDUALIZED 
INSTRUCTION  AN''  COMPUTER-BASED  INSTRUCTION 

Student  tr  g time  in  courses  can  be  reduced  without 
resort  to  oompucer-based  Instruction,  e.g.,  by  reducing  the 
amount  of  material  to  be  mastered  in  courses,  increased  re- 
liance upon  on-the-job  training.  Improved  conventional  instruc- 
tion, and  by  Individualized  instruction  (which,  by  definition, 
excludes  computer  support).  It  is  far  beyond  the  scope  of  this 
paper  to  consider  all  of  these  possibilities.  However,  we  found 
some  data  on  the  amount  of  student  time  saved  when  the  same 
courses  are  given  by  individualized  instruction  and  by  computer- 
assisted  or  computer-managed  instruction,  compared  in  all  cases 


"Practical  problems  in  the  Implementation  of  individualized 
instruction.  Navy  Personnel  Research  and  Development  Center, 
San  Diego,  California  (draft). 

Instructors 1 attitudes  towards  computer-managed  instruction. 
Navy  Personnel  Research  and  Development  Center,  San  Diego, 
California  (draft). 
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to  conventional  Instruction.  The  essential  issue  concerns  the 
benefit.  In  terms  of  additional  student  time  saved,  when  com- 
puter support  Is  given  to  individualized  instruction  (without 
computer  support) . Data  on  12  courses  are  summarized  in  Table 
9 and  shown  in  Pig.  6. 

TABLE  9.  AVERAGE  AMOUNT  OF  STUDENT  TIME  SAVED  BY 
INDIVIDUALIZED  INSTRUCTION  AND  CAI  OR  CMI  IN  THE 
SAME  COURSES,  COMPARED  TO  CONVENTIONAL  INSTRUCTION 


NO.  Of 

Average  Amount  of  Student  Time  Saved 

Course* 

CAI 

B 

64  V. 

6S% 

• 

7 

51% 

• 

51% 

Individualized  instruction  saves  large  amounts  of  student 
time  otherwise  required  by  conventional  instruction  (average 
savings  of  50  percent  or  more  in  these  samples).  The  addition 
of  CAI  to  five  individualized  courses  produced  additional  aver- 
age time  savings  of  5 percent;  the  addition  of  CMI  to  seven 
courses  produced  no  additional  time  savings.  Again,  no  signifi- 
cance can  be  given  to  the  differential  time  savings  observed  by 
adding  CAI  or  CMI  to  individualized  instruction  because  different 
courses  were  used  in  each  comparison. 

These  data  do  not  necessarily  imply  that  the  addition  of 
computer  support  to  individualized  instruction  is  not  cost- 
effective.  That  would  depend,  in  each  case,  on  whether  the  in- 
cremental costs  of  computer  support  are  offset  by  cost  reductions 
in  other  areas,  such  as  for  the  number  of  instructors  and  support 
personnel,  administrative  services,  and  other  factors. 
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FIGURE  6.  Amount  of  Student  Time  Saved,  Compared  to  Conventional  Instruction, 
by  Individualized  Instruction  and  by  CAI  or  CMI  on  the  Same  Courses 


Little  attention  has  been  given  to  the  benefits  associated 
with  different  types  of  computer-based  support  of  Individualized 
Instruction.  For  example,  from  early  AIS  data.  Student  Progress 
Management  saved  an  average  of  9 percent  of  student  time  in  four 
courses  and  Individualized  Instructional  Assignment  saved  an 
additional  3 percent  in  one  course  and  none  in  three  others 
(McDonnell  Douglas  Astronautics  1977a).  Only  student  time  sav- 
ings were  considered  in  these  reports  and  no  attention  seems  to 
have  been  given  to  other  possible  benefits  of  computer-support 
to  instruction. 

G.  SUMMARY 

1 . Effectiveness  of  CAI  and  CMI 

The  effectiveness  of  CAI  and  CMI  has  been  evaluated  in 
many  different  types  of  courses  in  military  training,  e.g.  , 
electronics,  vehicle  repair,  and  inventory  management.  These 
courses  include  both  knowledge  and  performance-oriented  skills. 
The  effectiveness  of  CAI  and  CMI  cannot  be  compared  directly 
because  in  no  case  was  the  same  course  given  by  both  of  these 
methods  of  instruction. 

2 . Student  Achievement 

Student  achievement  at  school  is  about  the  same  for  CAI, 
CMI,  and  conventional  instruction.  Some  evaluations  show  that 
student  achievement  with  CAI  is  superior  to  that  with  conven- 
tional instruction  but  these  differences  are  Judged  not  to  have 
practical  significance. 

3.  T Ime  Savings:  Computer-based  Instruction  vs.  Conventional 
Instruction 

Computer-based  instruction  appears  to  save  about  one-third 
of  the  time  required  by  students  to  complete  courses  given  by 
conventional  instruction.  There  is  a wide  variation  in  the 


amounts  of  time  reported  as  saved  in  experiments  but  research 
has  not  addressed  this  issue.  Based  on  experience  gained  in  the 
AIS  and  Navy  CMI  systems,  the  amounts  of  student  time  saved  ap- 
pears to  increase  over  time;  the  amounts  of  student  time  saved 
also  fluctuates  from  month  to  month. 

4 . Time  Savings:  Computer-based  Instruction  vs.  Individualized 
Instructl  on  ~~ 

About  the  same  amounts  of  student  time  are  saved  when  the 
same  courses  are  given  by  individualized  instruction  without 
computer  support  or  by  CAI  or  by  CMI. 

5.  Student  Attrition 

Student  attrition  for  academic  reasons  appears  to  increase 
slightly  when  CMI  replaces  conventional  instruction,  based  on 
experience  with  the  AIS  and  Navy  CMI  systems.  The  possibility 
that  these  increases  may  be  due,  in  part,  to  changes  in  student 
quality  and  to  other  factors  has  not  been  examined. 

6 . Student  and  Instructor  Attitudes 

Student  attitudes  to  CAI  and  CMI  tend  to  be  favorable. 

The  attitudes  of  instructors  appear  unfavorable  to  CAI  and  CMI 
in  comparison  to  conventional  instruction,  but  this  finding  is 
based  on  limited  dat«.  The  role  of  Instructors  in  CAI  and  CMI 
has  received  little  attention,  both  as  to  collecting  more  re- 
liable information  about  their  attitudes  and  to  preparing  them 
for  handling  these  new  methods  of  instruction. 


IV.  COSTS  OF  MILITARY  INSTRUCTION 


Surprisingly  little  appears  to  be  known  about  either  the 
determinants  or  composition  of  the  costs  of  Instruction.  Each 
Service  has  a system  for  reporting  the  costs  of  training  courses 
for  such  purposes  as  development  of  estimating  relationships  and 
evaluation  of  proposed  training  program  alternatives;  their 
actual  use  appears  to  be  limited  to  providing  the  average  total 
(bottom-line)  costs  of  individual  courses  for  such  purposes  as 
setting  reimbursement  rates  for  training  foreign  students  and 
those  from  other  Services.  This  information  sheds  no  light  on 
questions  of  why  training  costs  are  what  they  are  or  how  they 
would  change  in  response  to  changes  in  training  courses,  such  as 
the  method  of  instruction  or  the  content  of  course  materials. 

In  general,  it  may  be  said  that  neither  the  detailed  data  on 
training  costs  nor  the  methodology  for  analysis  (as  opposed  to 
accounting)  of  training  costs  have  been  developed.  The  develop- 
ment and  maintenance  of  a data  base  on  the  costs  of  military 
instruction  are  far  from  cost-free,  and  the  question  of  what  data 
"should"  be  collected  can  only  be  assessed  by  further  questioning 
their  cost  and  worth  of  such  data  in  supporting  cost-effective- 
ness analyses  of  military  training. 

This  chapter  addresses  two  problems  associated  with  the 
collection  of  data  on  the  costs  of  training.  The  first  is  to 
examine  how  the  organizational  structure  of  formal  military 
training  affects  the  collection  of  relevant  cost  data.  The 
second  is  to  assess  data  on  costs  of  training  that  have  been 
developed  in  various  experimental ' programs  and  studies  of  com- 
puter-based instruction. 
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A.  COLLECTION  OF  COST  DATA 


The  resources  required  for  instruction  are  structured  simi- 
larly in  both  military  training  and  civilian  education.  For 
example,  consider  a school  which  offers  a few  courses  in  a limi- 
ted number  of  subjects  and  grades.  Resources  can  be  grouped 
according  to  type  or  funotion.  They  can  also  be  grouped  accord- 
ing to  where,  within  the  organizational  structure,  these  resources 
are  directly  applied  and  identified.  This  two-way  grouping  de- 
scribes a matrix,  as  shown  in  Fig.  7.  Some  resources,  such  as 
for  instructors  and  certain  equipment,  are  dedicated  to  a par- 
ticular course  and  their  costs  can  be  associated  directly  with 
that  course.  Other  resources,  such  as  for  facilities  and  other 
equipments,  serve  a number  of  courses  in  common.  Requirements 
for  the  use  of  common  resources  may  vary  widely  between  the 
courses  offered,  with  the  result  that  the  costs  of  different 
courses  may  vary  significantly  in  ways  that  are  concealed. 
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Pragram  Devatapuuirt 

Program  Delvery:  instructors 

Equipment 

Supples  end  CensumaMes 

FacNOes 

Program  Management 

■iiilitl 

: y 
' - % 

/ 

: - -v. 'W/. 
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- ' / 
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FIGURE  7.  Resource  matrix 
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1 . Organization  of  Military  Training 

At  a small  school,  the  use  of  common  resources  by  particular 
courses  are  readily  visible  and  thus  their  costs  may  be  easily 
associated  with  different  courses.  For  example,  the  cost  of 
using  media  devices  may  be  allocated  systematically  to  indi- 
vidual courses  by  keeping  simple  records  of  purchase  and  main- 
tenance costs  and  the  hours  of  use  in  different  courses.  In 
current  military  training,  the  relationships  between  courses, 
resource  use,  and  costs  of  training  are  complex  and  obscured  by 
the  large  size  of  training  establishments  and  the  manner  in  which 
they  are  organized.  Some  types  of  resources  are  expensive  to  use 
compared  with  others,  and  some  courses  are  expensive  to  offer 
compared  with  others;  the  records  required  to  trace  the  use  of 
resources  by  particular  courses  are,  themselves,  complex  and 
expensive.  This  is  the  heart  of  the  problem  faced  in  the  collec- 
tion and  evaluation  of  data  on  the  costs  of  training. 

A representative  organizational  structure  for  military 
training  is  shown  in  Fig.  8.  It  is  adapted  from  a "typical" 

U.S.  Air  Force  organization  as  shown  in  Hess  and  Kantar  (1977). 
The  critical  feature  is  the  deep  hierarchy  of  organizations  that 
support  and  manage  Individual  courses  and  lessons.  An  explana- 
tion for  this  degree  of  complexity  is  that  it  is  needed  to  attain 
an  efficient  scale  of  operation.  For  example,  instructional  ma- 
terial is  typically  associated  with  an  Individual  lesson  and 
instructors  may  be  best  utilized  as  specialists  in  a single 
course.  However,  some  training  resources,  especially  in  tech- 
nical training,  come  in  large  and  indivisible  units  that  may  be 
employed  efficiently  only  where  student  loadings  exceed  those  of 
individual  courses,  training  branches,  departments,  schools,  and 
possibly  the  total  student  population  of  a training  facility.  A 
conspicuous  example  of  a large  and  indivisible  training  input  is 
the  central  processor  of  the  PLATO  IV  system. 
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FIGURE  8.  Management  of  Instruction  according 
to  organizational  structure 
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The  formal  organizational  structure  extends  through  the 
training  brashes  that  are  responsible  for  a number  of  closely 
related  courses.  The  course  (and  ultimately  the  Individual 
lesson)  is  the  elemental  unit  of  training  and  the  generator  of 
all/ training  costs;  resource  expenditures  necessary  for  its  con- 
duct occur  at  each  of  the  higher  levels  of  the  structure  (includ- 
ing the  training  branches),  and  the  expenditures  incurred  at  any 
higher  level  node  may  support  the  whole  range  of  instruction  and 
support  activities  beneath  it.  In  essence,  the  whole  structure 
above  the  course  level  is  equivalent  to  the  columns  labelled  '"all 
coursee"  in  a small  school  (Fig.  7).  In  a large  training 
organization,  the  visibility  of  who  provides  what  for  whom  and 
who  receives  what  from  where  is  quickly  lost.  The  relationships 
among  units  at  various  levels  in  military  training  organizations 
are  too  complex  to  be  traced  by  simple  bookkeeping  procedures. 

In  the  absence  of  extensive  data,  the  relationships  between 
instruction  and  the  expenditure  of  resources  for  Instructional 
support  in  military  training  cannot  be  determined.  The  costs  of 
supporting  particular  courses  can  be  estimated  only  by  highly 
arbitrary  allocation  rules  that  may  bear  little  resemblance  to 
the  true  sources  of  cost.  Discovering  these  relationships  is  an 
essential  ingredient  of  the  capability  to  evaluate  training  coBts. 

2 . Alternative  Mays  of  Collecting  Cost  Data 

More  precise  and  detailed  cost  data  are  needed  to  support 
cost-ef fectiveness  analyses  of  alternative  methods  of  instruction 
in  military  training.  Training  cost  data  can  be  collected  in 
three  ways  that  differ  widely  in  scope  of  effort  and  cost.  The 
first  is  to  collect  cost  data,  through  a formal  reporting  system, 
for  all  training  courses  offered  by  a Service,  including  all 
costs  for  conducting  and  supporting  training  programs.  The  ex- 
tent, detail,  and  identification  of  cost  data  would  have  to  be 
greater  than  that  provided  by  current  systems,  especially  with 
regard  to  training  support  functions.  Since  a course  may  employ 
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several  methods  of  Instruction  (conventional.  Individualized, 
and  computer-based),  data  would  have  to  be  identified  with  small 
units  of  instruction,  possibly  individual  lessons,  if  the  system 
were  to  provide  information  on  the  relative  costs  of  different 
methods  of  instruction. 

The  second  method  is  to  collect  information,  also  through 
a formal  reporting  system,  for  a sample  of  organizations  that 
conduct  and  support  training.  It  is  recognized  that  the  military 
Services  have  rarely  applied  sampling  techniques  to  collect  cost 
data.  Basic  questions  regarding  the  extent  and  duration  of  samp- 
ling would  have  to  be  resolved  before  a program  for  collecting 
data  could  be  designed  or  procedural  problems  addressed.  The 
collection  system  could  not  disturb  existing  management  and 
data-collectlon  systems,  and  it  would  have  to  be  implemented  in 
a manner  that  would  not  distort  either  the  level  or  structure  of 
training  costs  in  the  activities  to  be  observed. 

The  third  alternative  is  to  perform  ad  hoo  studies  of 
organizations  that  conduct  and  support  training  programs.  Costs 
of  ad  hoo  study  should  be  lower  than  those  of  formal  reporting 
systems,  and  it  offers  the  advantage  of  flexibility.  Studies 
can  address  specific  topics  of  high  interest  and  focus  attention 
on  cost  and  non-cost  responses  to  systematic  changes  in  study 
parameters,  e.g .,  student : instructor  ratios  or  length  of  train- 
ing day;  data  collection  may  be  tailored  to  the  questions  ad- 
dressed and  organizations  examined.  Ad  hoo  study  programs  also 
have  problems.  A principal  one  is  to  maintain  financial  support 
that  is  adequate  for  pursuing  a coherent  and  on-going  program 
and  for  providing  data  in  a timely  fashion  and  depth  that  recog- 
nizes the  full  extent  and  sources  of  all  training  costs. 

B.  AVAILABLE  DATA  ON  COSTS  OF  INSTRUCTION 

This  section  summarizes  the  data  we  were  able  to  find  on 
the  costs  of  computer-based  and  other  methods  of  instruction; 
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the  detailed  data  are  presented  in  Appendix  E.  Here,  we  discuss 
the  adequacy  of  these  data  for  assessing  the  costs  of  different 
methods  of  instruction.  We  have  excluded  data  that  were  not 
described  well  enough  to  be  interpreted  with  confidence.  All 
data  are  shown  as  they  were  found  in  the  literature.  No  investi- 
gations of  their  validity  have  been  performed,  and  no  adjustments 
have  been  made  for  changes  in  price  levels. 

Table  10  shows  the  number  of  sources  of  data  on  costs  of 

instruction,  arranged  according  to  method  of  instruction  and  type 

of  resource.  The  cost  data  come  either  from  eight  experiments 

(identified  in  Table  1,  p.  27)  or  from  other  sources  (identified 

in  Appendix  E) ; shaded  cells  indicate  that  cost  data  are  not 

applicable;  blank  cells  indicate  that  relevant  cost  data  are 

•* ' •••..  ... 

not  available. 

Table  10  shows  that  there  are  few  sources  of  data  on  the 
costs  of  Instruction  in  military  training;  especially  notable  is 
the  absence  of  information  on  the  costs  of  conventional  instruc- 
tion. The  troublesome  nature  of  this  is  obvious  when  one  con- 
siders that  the  cost-effectiveness  of  CAI  or  CMI  must  be  compared 
to  that  of  some  other  method  of  instruction,  generally  conven- 
tional instruction.  Two  other  important  omissions  concern  Program 
Design  and  Program  Management.  The  design  of  instructional  pro- 
grams (Program  Design)  may  be  a significant  cost  item  because  of 
current  emphasis  on  Instructional  System  Development  in  all 
military  Services,  but  Its  costs  appear  to  be  either  ignored  or 
combined  with  those  of  instructional  materials.  Since  Program 
Design  is  independent  of  the  method  of  instruction  and  may  have 
its  own  impact  on  instructional  costs,  its  cost  should  be  sepa- 
rated from  the  cost  of  instructional  materials.  Program  Manage- 
ment may  be  a major  cost  item  because  of  the  large  and  highly 
structured  organizations  in  which  military  instruction  takes 
place,  and  this  cost  may  differ  between  instructional  methods. 
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TABLE  10.  NUMBER  OF  SOURCES  OF  DATA  ON  COST  OF  INSTRUCTION, 
ACCORDING  TO  METHOD  OF  INSTRUCTION  AND 
RESOURCE  TYPE  OR  FUNCTION  (See  Note) 


METHOD  OF  INSTRUCTION  


CONVENTIONAL  MOtVtOUALIZED  COMPUTER-BASED  WSTRUCTWN 
INSTRUCTION  INSTRUCTION  PLATO  IV  I NAVY  CM  | OTHER* 


RESOURCE  (TYPE  OR  FUNCTION) 


Program  Development 
Program  Design 

Instructional  Materials:bConventional  Instruction 
Indlvldiulzed  Instruction 

Programming 

First  Unit  Production*- 
Computer-Based  Instruction 
Programming 
Coding 


Program  Delvery 
Instruction:  Instructors 

Instructional  Support  Personnel 
Equipment  and  Services:'  Laboratory  (ind.  simulators) 
Media  Devices 
Computer  Systems 
Communications 
_ Materials  (Incl.  Consumables/ 

‘ FaclUtlesB 

Program  Management  and  Administration 
Student  Personnel:  Pay  and  Allowances 

Others  (PCS.  TDY,  tte.  ' 


NOTE:  Shtdud  crti  an  ra<  ipplcibla.  Blank  cafe  Mlcala  that  ntavant  cart  data  an  nat  araeawa. 

’Includes  Ttccrr,  IBM  1500.  LTS-3,  GETS,  and  an  experiments!  lMpboatd  system, 

•hncludei  revision. 
eMester  copy. 

dM  dlnct  paraonnel  not  Included  In  othar  cstegorles. 

■Includai  al  hsrdwere  nlatod  costa:  Initial  (Including  Insulation  and  eneckovt).  modfflcjtton.  and  replacement:  aptratton  and  maintenance; 
Inara  and  utar  toot:  computer  system  software;  ate 
•includes  coplat  o<  Irstniclionsl  malortalt  (bookt,  caurtawan  coplti,  «t). 

Ittructurss,  fixtures,  and  turnlihlngi. 

•’leDorilory  equipment  and  madia  device*  are  appicaMe  to  al  malhodt  ot  Inunction  (iscept  whore  simulated  In  CAJ  systems).  and  there 
it  no  reaton  why  cotta  ot  Ihetr  uto  would  dlttar  with  mothod  ot  Inilructton. 

I Parmanenl  change  of  ttatlon.  temporary  duty. 


The  sample  of  data  on  costs  of  training  presented  here  is 
meager  by  a-y  standards.  We  cannot  begin  to  explain  the  range 
of  costs  that  has  been  observed,  and  we  feel  uncertain  as  to  the 
feasibility  of  a generalized  parametric  approach  to  estimating 
the  costs  of  training.  Extensive  further  efforts  to  collect  and 
Interpret  data  on  the  costs  of  training  would  be  required  before 
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the  value  of  parametric  analysis  of  training  costs  could  be 
Judged  or  a general  model  could  be  formulated. 

The  following  sections  discuss  the  cost  data  that  we  were 
able  to  find  and  use. 

1 . Program  Development:  Program  Design 

No  data  were  found  on  these  costs  of  instruction. 

2 . Program  Development:  Instructional  Materials  for 
Conventional  Instruction 

No  data  were  found  on  these  costs  of  instruction. 

3 . Program  Development:  Instructional  Materials  for 
Individualized  Instruction 

Development  of  instructional  material  for  individualized 
instruction  encompasses  two  distinct  and  separable  functions: 
the  programming  (authorship)  and  the  production  of  first  units 
(master  copy)  of  media  material  (including  printed  text).  The 
cost  data  we  found  are  shown  in  Table  11.  The  most  notable 
feature  of  resource  requirements  is  the  great  variation  asso- 
ciated with  both  functions. 

With  regard  to  programming  (authorship),  the  available  in- 
formation shows  two  widely  separated  values;  i.e.,  40  and  280 
man-hours  per  instructional  hour;  the  source  provides  no  dis- 
cussion for  the  large  difference  in  values  for  the  two  media. 

With  respect  to  first  unit  production  (£.«,,  master  copy), 
costs  of  different  media  range  from  $12,000  for  an  hour  of  sound 
motion  picture  or  TV  tape  to  a tew  hundred  dollars  for  printed 
text  and  silent  slide  or  film-strip  (assuming  30  pages  or  frames 
per  hour).  For  the  same  media,  costE  range  from  near  $500  to 
$2,000  per  instructional  hour  for  sound-slide  and  $10  to  $400 
for  printed  illustration,  and  the  literature  provides  no  expla- 
nation for  these  clifferenres. 


TABLE  11.  INDIVIDUALIZED  INSTRUCTION:  INSTRUCTIONAL 
MATERIALS  DEVELOPMENT,  REQUIREMENTS,  AND  COSTS 


RESOURCE 

IMT  Of  MEASURE 

MUMPER  Of 
SOURCES 
OF  DATA 

ESTMATES 

REFERENCES 

Programming 

Sound  Motion  Picture  or  TV 

Man-heurs/ln  ttructlonai  Hour 

1 

280 

U.S.  Army  Ordnance  Center  and 

School  (1975) 

SoundSMe 

Man-hours/In ttructlonai  Hour 

1 

40 

U.S.  Army  Ordnance  Center  and 

School  (1975) 

First  Unit  Production 

Sound  Motion  Picture  or  TV 

Oolars/inttrucISonal  Hour 

2 

$6,000-12,000 

Hess  and  Kantar  (1977);  U.S.  Army 

Ordnance  Center  and  School  (1975) 

Sound-SMo 

0olare/ln ttructlonai  Hour 

2 

$ 500-  1,925 

Hess  and  Kantar  (1977);  U.S.  Army 

Ordnance  Center  and  School  (1975) 

Stent  Motion  Picture 

Oofiars/fcistnictfonal  Hour 

1 

S 10,200 

Hess  and  Kantar  (1977) 

Silent  SMe  (or  Film  Strip) 

Reala 

Dofiars/SMe 

1 

S 1 

Heat  and  Kantar  (1977) 

lustration 

Dofiart/SMe 

1 

$ 3-115 

Hess  and  Kantar  (1977) 

Printed 

Text 

Dolors /Page 

1 

$ 7 

Hess  and  KanUr  (1977) 

lustration 

MUrt/Page 

1 

$ 11  -430 

Hess  and  Kantar  (1977) 

‘ Audio 

Dolars/kistnictional  Hour 

1 

$ 180 

Hess  and  KanUr  (1977) 

Combined  or  Not  Specified 

Man-haura/bistructlonal  Hour 

2 

40  - 200 

Dolman,  DeLeo  rt  d (1977); 

MMtoton.  Papettl,  and  Michel  (1974) 

Oolart/lnstnicttena)  Hour 

2 

$1,130-15,800 

PMcyn.  Baudhuln.  Brekkj  rt  d (1977); 

TemUn,  Connody  rt  d (1975) 

3-28-79-31  4-11-79 


With  such  wide  ranges,  the  usefulness  of  cost-effectiveness 
analysis  may  be  questioned.  One  of  its  principal  applications 
lies  in  identifying  and  separating  promising  from  unattractive 
alternatives  early  in  course  design,  i.e.  , before  significant 
resources  have  been  committed.  Much  of  its  value  in  this  role 
is  lost  if  a significant  course  design  effort,  to  identify  media 
mixes,  is  required  to  provide  initial  assessments  of  alterna- 
tives. This  situation  argues  for  the  application  of  course 
design  procedures  such  as  MODIA  ( Carpenter-Huf fman , 1977)  and 
TECEP  (Braby,  Henry,  et  al . , 1975)  at  command  levels  where  rele- 
vant policy  is  formulated  and  decisions  are  made  (see  Lackland 
AFB , 1978). 
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4 . Program  Development:  Instructional  Materials  for 
Computer-Based  Instruction 

In  computer-based  instruction,  development  of  instructional 
materials  also  encompasses  two  distinct  functions.  The  first  is 
programming  or  authorship,  similar  to  individualized  instruction. 
The  second,  coding,  organizes  the  material  into  a form  suitable 
for  machine  processing.  Resource  requirements  for  close  to  1,000 
hours  of  instruction  were  cited,  and  large  variability  is  again 
present  (see  Table  12). 

TABLE  12.  COMPUTER-BASED  INSTRUCTION:  REQUIREMENTS  FOR 
INSTRUCTIONAL  MATERIALS  DEVELOPMENT 


METHOD  OF 
MSTHUCTI0H 

SYSTEM 

MSTHUCnONAL 

HOOKS 

DEVELOPED 

MANHOURS  PER  MSTRUCTI0HAL  HOUR 

REFERENCES 

TOTAL 

39.0 

27  to  248 

50  to  467 

77  to  714 

Hurlock  a Slough  (1976) 

156 

Kribt  (1976) 

400 

Kiibi  (1976) 

PLATO  IV 

284 

U.S.  Army  Ordnance  Center  a School  (1975) 

20.0 

100  & 200 

Dafenan,  DeLeo  af  d (1977c) 

32.0 

141 

81 

222 

Hbnwfch  (1977) 

CAI 

315.0 

80 

Grtmoa  (1975) 

■■ : ■ ■ ■ 

■ I 

Kith*  (1976) 

Ticcrr 

■s 

Kribt  (1976) 

150 

96 

246 

Hhnwieh  (1977c) 

LTS-3 

30.0 

175 

Keealer  AFB  (1973) 

WM-1500 

3.5 

356 

119 

475 

L it 

Unapecfflod  CAI 

Unknown 

- 

150  a 200 

Middleton.  Papottl  a Michel  (1974) 

100 

10 

110 

Carton.  Graham  rt  d (1975) 

CM 

Navy  CM 

30  8 60 

Hanien,  Rosa  id  (1975) 

Unknown 

(293)* 

(25)* 

(318)* 

Polcyn,  BaudhuJn  #r  d (1977) 

*l«l»i»iiti  m |hn  la  torn  * Man  p«  laslnic««ul  txririlmlM  CNnrM  n tkt  usampON  * $10  par  Iw  tor  ariKvr 


For  CAI,  authoring  is  cited  as  ranging  between  about  30  and 
360  man-hours  per  instruction  hour,  and  coding  between  50  and 
^70  man-hours.  These  ranges  cannot  be  attributed  to  extraneous 
factors,  such  as  differences  in  the  way  expenditures  are  accounted 
for,  since  close  to  the  total  range  of  variation  was  noted  in  one 
study  (Hurlock  and  Slough,  1976)  summarizing  eight  experimental 
programs  performed  by  the  same  organization  and  utilizing  the 
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same  CAI  system  in  roughly  the  same  time  period.  These  data  are 
also  insufficient  for  attributing  different  programming  and  coding 
requirements  to  different  CAI  systems  and/or  alternative  instruc- 
tional strategies.  For  CMI,  the  variation  is  similar;  from  less 
than  30  to  290  man-hours  for  authoring  and  10  to  25  man-hours  for 
coding. 

5.  Program  Deli  very:  Instruction,  Including  Instructors 
and  Instructional  Support  Personnel 

Only  three  sources  provide  data  on  costs  of  instructors  and 
instructional  support  personnel.  A few  citations  to  student 
personnel  cost  are  also  included  here.  The  data  are  limited  to 
pay  and  allowance  rates,  student : instructor  ratios,  and,  in  one 
source,  instructional  and  indirect  support  personnel  ratios  (see 
Table  13). 

Personnel  expenditures  are  considered  to  account  for  the 
bulk  of  training  costs,  and  those  associated  with  students,  in- 
structors, and  with  instructional  support  personnel  must  be 
aseumed  to  be  significant.  In  the  absence  of  other  changes, 
decreases  in  course  lengths  ( e.g .,  associated  with  an  introduc- 
tion of  computer-based  instruction)  would  result  in  lower  student 
loads  and  proportional  decreases  in  instructor  personnel. 

6 . Program  Delivery;  laboratory  Equipment 

No  data  were  found  on  these  costs  of  instruction. 

7 . Program  Delivery;  Media  Devices 

Estimating  costs  of  use  of  media  devices  entails  extensive 
training  course  specification  similar  to  that  associated  with 
production  of  master  copies  of  media  materials.  A large  variety 
of  devices  is  available  at  widely  differing  costs.  Equipment  is 
long-lived  and  can  be  shared  by  different  courses.  Representa- 
tive cost  ranges  are  shown  in  Table  14. 

One  comprehensive  catalogue  (The  Audio-Visual  Equipment  Di- 
rectory, 1970,  published  by  the  National  Audio-Visual  Association) 
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TABLE  13.  INSTRUCTOR  AND  SUPPORT  COSTS  FOR  PROGRAM  DELIVERY, 
COMPUTER-BASED  INSTRUCTION 


DATA  ITEM 

SOURCE 

Carson, 
Graham, 
Harding 
$t  d 
(1975) 

Hansen, 

Ross, 

Bowman,  & 

Thurmond 

(1975) 

Crawford, 
Hurtock, 
Padillo,  & 
Sassano 
(1976) 

System 

Navy  CMI 

Navy  CMI 

PLATO  IV 

Change  in  Ratios 

Students : Instructors 

10:1  to 

7.5:1  to 

a 

16:1 

9.0:1 

Students  to  Instructional 

- 

Unchanged 

- 

(Direct)  Support 

at  24:1 

Students  to  Indirect 

- 

Unchanged 

(Base)  Support 

at  12.5:1 

Pay  and  Allowance  Rates 

Students 

$5,899 

•mmmsm 

Instructors 

9,697 

Instructional  Support 

-- 

•• 

- 

Indirect  Support 

-- 

12,400 

-- 

*Cinnot  be  txpmtml  In  thete  term*.  The  net  remit  wu  to  elminite  the  tingle  initructor-hour  container!  in  a 9-hour 
training  tegmenl. 

^Applet  to  students,  instructors,  and  instructional  support  personnel. 

cThe  $61,000  figure  Is  described  as  Mllet  cost  and  includes  a variety  of  personnel  support  Items  over  and  above 
pay  and  adowances,  «.£.  command  and  administration,  dependent  school  costs,  recruiting  costs,  reenlistment 
bonuses,  and  retirement;  students  and  Instructors  were  pitots. 
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lists  nearly  1,C00  presentation  and  presentation  control  devices 
classified  into  over  50  types  of  commercially  available  equipment. 
The  size  of  some  types  of  equipment  (e.y.,  motion  picture  pro- 
lectors)  varies  between  that  suitable  for  a large  auditorium  to 
that  used  by  an  individual.  The  purchase  cost  of  some  types  of 
equipments  will  vary  by  more  than  an  order  of  magnitude,  depend- 
ing upon  size  and  features.  Selections  of  equipments,  then, 
require  specifications  of  both  the  type  of  device  and  the  en- 
vironment in  v/hich  course  materials  will  be  presented. 
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TABLE  14.  MEDIA  DEVICES:  UNIT  COSTS  AND  OTHER  INFORMATION3 


MEOIA  DEVICE 

RANGE  OF 
MT1AL  COST 
(DOLLARS) 

HESS  AM)  KANTAR 

(1977) 

UFE 

SPAN 

(YEAR) 

HESS  AND  KANTAR 

(1977) 

MEAN  TIME  BETWEEN  FAILURES  (H0UR8) 

HESS  & KANTAR 

(1977) 

MCDONNELL  DOUGLAS 
(1977a) 

Seund  Mevt*  Prefecture 

175-1,000 

« 

90-110 

197 

Vldeetape  Recerdera/Ptayer* 

600-8,000 

5 

463 

Seund  Slde/Sttp  Pre)acte« 

100-1,000 

820 

5 ■till  MCVN  ITtftCTtrs 

150-250 

Slmt  sadt/SMp  Prefectere* 

25-900 

6-10 

90-150 

377-3,711 

Randem  Acces*  8ld«  Prelectors 

500-2,000 

Mcrtffira/RclM  Ruder* 

80-800 

2,785 

Audietape/Dtsc  Player* 

30-325 

2,783 

Taachiog  Machinal  (Individual) 

‘ 

Audio  Visual 

Rate  Central 

230-1,000 

Constant  Control 

1950 

Visual 

Rate  Control 

140-380 

Constant  Control 

220-1,200 

Audio 

Rate  Control 

190-470 

TV  Monitor 

2,315 

Headsdt 

27,240 

*EicM*«  MHpunti  tM  Urp  l«r  mm  In  ImUvMuiI  claurMmt.  CmIi  in  l«  CMnnwrcW  qutty  nqulpnitmt. 

SkM*i  tv  trim  an  pnftctan. 

J-21-79-34  4-18-79 


Complicating  the  problem  of  determining  the  cost  of  use  is 
the  fact  that  media  devices  are  typically  long-lived  and  repair- 
able assets,  are  generally  portable,  and  can  be  employed  in  a 
number  of  classes  at  a number  of  locations.  The  cost  of  use, 
then,  depends  upon  anticipated  lifespan,  failure  rates,  and  re- 
pair costs,  in  addition  to  purchase  cost  and  rate  of  usage.  The 
cost  attributed  to  an  individual  course  also  depends  upon  whether 
required  equipments  are  currently  on  hand  and  available  for  use 
(i.e.,  currently  unemployed).  The  latter  point  implies  that  one 
must  also  consider  current  inventories  and  usage  rates  of  other 


courses  in  determining  the  cost  ol  using  media  devices. 


72 


I 


8.  Program  Delivery:  Computer  Systems 
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Substantive  cost  information  is  available  on  five  computer 
hardware  systems--IBM  1500,  PLATO  IV,  TICCIT,  GETS,  and  Navy  CMI. 
These  five  represent  a wide  range  of  capabilities  in  terms  of 
the  number  of  terminals  supported  by  a single  central  processor. 
They  also  differ  widely  in  terms  of  contractual  arrangements 
under  which  they  have  been  obtained  (purchase,  lease,  or  a com- 
bination). This  makes  interpretation  of  the  information  impre- 
cise and  subject  to  considerable  qualification. 


The  detailed  information  that  we  were  able  to  compile 
appears  in  Appendix  E.  Table  15  summarizes  the  costs  of  these 
systems  expressed  in  the  following  ways: 


• Central  processing  unit 

• Terminal 

• Total  system  hardware 

• System  cost  per  terminal 

• System  cost  per  student -hour 


These  data  should  be  accepted  primarily  for  illustrative  pur- 
poses; any  contemplated  application  would  need  current  data  on 
systems  configured  to  particular  specifications  of  interest. 

Three  principal  resource  categories  can  be  associated  with 
computer  system  use:  (1)  the  hardware,  (2)  its  operation,  and 
(3)  its  maintenance.  Little  information  is  available  on  either 
maintenance  or  operations.  Maintenance  estimates,  based  on  the 
IBM  1500  and  PLATO  IV,  range  from  15  to  35  percent  of  hardware 
purchase  cost  over  a 5-year  period;  the  lower  limit  Is  asso- 
ciated with  the  IBM  1500.  However,  all  IBM  1500  systems  in  the 
sample  were  leased,  and  the  15  percent  figure  is  based  on  amor- 
tizing lease  charges  over  a 5-year  period.  Operating  cost  may 
vary  greatly  as  a function  of  the  user's  organization,  and 
sketchy  information  on  IBM  1500  use  indicates  such  a variation, 
ranging  between  5 and  50  percent  of  annual  lease  costs. 


1 


1 
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TABLE  15.  COSTS  OF  COMPUTER  SYSTEMS  HARDWARE 


p 

Piaster  System 

Cm M 
PnCMMT 

Cwt 

(TOmimOi) 

Terminal' 

UattCest 

(TkSUSJfldt) 

System 

Hardwire 

Cad 

(Tbeusands) 

System 
HarOwan  Cast 

Par  Terminal 
(Tlteu  sands) 

Syd»« 
Hardware  Cast 

Par  Student- 
Hew* 

KM  IBM 

32  TtnMufc* 

$ 000 

$25 

$2.40 

PLATON 

1,000  TwMMr* 

$0,000 

$0.7 

10,700 

11 

1.4#- 

CM 

TICCIT 

32  TanMtate* 

700 

2.) 

27 

2.66 

04  Tenatoab 

070 

2.0; 

l 

16 

1.64 

121  Tertaub 

•70 

2.0 

. , 

10 

1.04 

• 1 

SETS 

0m  Terminal 

- 

- 

34 

34 

3.40 

Navy  CM 

0,000  Students' 

2,300 

14.3 

4,020 

34 

0.07 

CM 

10,000  Students* 

2,300 

14.3 

0,000 

22 

0.04 

*1.MI  tan  pm  Wrtauf  pm  ytm  Mr  I pan. 

, HkMm  ■rttaniti.  lata  aa  laaaa  rate*  mi  mitiKi  • pmpmrnt  ever  a pmM.  1M7, 1*72,  1*77. 
cCa*M  Data  Cereweaee  pmtatm.  treat  private  tweelniea  4aM  14  Aaaeet  1(71. 

‘leaed  aa  728  artaa  Untari  caaaWat 
'Mirtaat  ytataiaa.  treat  private  fataataatcrtaa.  1171. 

*12t  tanalata  at  M tta4aata  pm  tanahul.  1*77. 

*320  lattaMta  at  M iMaatt  pm  lanaiaai.  1177. 

3-21-78-18 


System  hardware  costs  can  be  expressed  in  three  ways:  (1) 
system  procurement  cost,  (2)  cost  per  terminal  connected,  and 
(3)  cost  per  student-hour  (over  some  chosen  amortization  period). 
In  terms  of  system  procurement  cost,  a range  between  near  $35,000 
(the  stand-alone  GETS)  and  over  $10  million  (a  1,000-terminal 
PLATO  IV  system)  can  be  noted,  a factor  of  close  to  300  times. 

On  a per-terminal  basis,  though,  available  information  indicates 
an  Inverse  relationship  between  system  size  and  cost.  As  an 
example,  for  the  TICCIT  system,  the  per-terminal  system  cost  of 
the  32-terminal  configuration  is  close  to  two-and-one-half  times 
that  of  the  128-terminal  configuration.  This  information  indi- 
cates a substantial  economy  of  scale  for  larger  systems. 
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A more  meaningful  relationship  in  comparing  computer-based 
instruction  with  other  methods  is  the  cost  per  student-hour. 

This  cost  is  inversely  proportional  to  terminal  utilization  rates, 
to  system  life  span  and,  for  computer-managed  instruction,  to  the 
number  of  students  a terminal  can  accommodate.  Realized  costs 
per  student-hour  will  be  highly  sensitive  to  each  of  these.  An 
average  of  2,000  hours  per  year  per  terminal  is  a widely  cited 
target  value,  but  one  which  appears  difficult  to  attain.  Should 
it  prove  attainable,  and  assuming  a system  life  span  of  5 years, 
indicated  student-hour  costs  for  CAI  systems  range  between 
roughly  $1.00  (the  128-terminal  TICCIT  system)  and  $3.50 
(GETS).  The  lower  per  student-hour  cost  associated  with  large 
systems  implies  a large  initial  commitment  of  funds  (if  central 
hardware  is  purchased)  and  a large  commitment  to  CAI  with  the 
other  costs  and  risks  it  entails.  Assuming  that  each  CMI  termi- 
nal would  accommodate  50  students,  student-hour  costs  would 
appear  to  be  less  than  $0.10. 

Note  that  the  $3.50  associated  with  the  GETS  is  based  on 
information  that  is  several  years  old.  Systems  of  comparable 
capability.  Incorporating  recent  technological  advances  in  micro- 
processors and  data  storage  devices,  can  be  anticipated  to  cost 
considerably  less. 

9.  Program  Delivery:  Communications 

Communications  are  relevant  only  for  large  systems  where 
terminals  may  be  geographically  separated  from  central  pro- 
cessors. In  current  military  applications,  these  are  limited 
to  PLATO  IV  and  the  Navy  CMI  system.  Communications  have  been 
accomplished  through  two  modes--microwave  transmission  and  land- 
lines, but  microwave  transmission  has  received  too  little  dis- 
cussion to  allow  characterization  of  Its  costs  here.  The  rate 
schedule  for  communications  over  Commercial  long  lines,  as 
reported  in  two  studies,  is  shown  in  Table  16. 


TABLE  16.  RATE  SCHEDULE  FOR  COMMUNICATIONS  OVER 
COMMERCIAL  LONG  LINES 


DISTANCE 

INTERVAL 

(MILES) 

COST  PER  INCREMENTAL 

MILE  PER  MONTH 

AVERAGE  COST  PER  MILE 
PER  MONTH  (AT  LIMIT  OF 
DISTANCE  INTERVAL 

1-25 

$3.30 

$3.30 

26-1  CP 

2.31 

2.56 

101-250 

1.65 

2.01 

251-500 

1.15 

1.58 

>500 

0.83 

1.20* 

•At  1000  miles. 

Source:  Bal  and  Jamison  (1973),  and  Mdtieton,  Papetti,  and  Micliel  (1974). 
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1 For  land-lines,  a commonly  used  rule  of  thumb  is  $1.00  per 
mile  per  month  for  long  lines  (interstate),  but  line  distances 
of  greater  than  1,000  miles  are  required  before  costs  decrease 
to  this  level.  Rates  charged  government  agencies  for  lines 
leased  through  the  General  Services  Administration  are  typically 
stated  at  half  the  commercial  rate,  and  $0.50  per  mile  per  month 
is  the  value  commonly  used  in  military  studies.  The  significance 
of  communications  costs  for  a large  system  with  a central  com- 
puter can  be  appreciated  by  the  following.  At  the  $0.50  per 
mile  per  month  rate,  estimated  communications  costs  in  the  PLATO 
IV  experimental  programs  averaged  over  50  percent  of  computer 
rental  and  terminal  maintenance  costs. 

1 0 . Program  Delivery:  Materials 

No  data  were  found  on  these  costs  of  instruction. 
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1 1 . Program  Delivery:  Facilities 


During  time  periods  in  which  the  military  services  are  not 
expanding,  the  need  for  additional  facilities  and  furnishings 
should  be  a minor  consideration  in  instructional  costs.  Cases 
where  such  costs  would  be  incurred  would  be  limited  to  the  intro- 
duction of  new  training  courses  (as  might  accompany  the  intro- 
duction of  new  operational  equipments)  and  major  changes  in  the 
way  instructional  material  is  presented.  In  both  cases,  re- 
quirements might  be  levied  for  modifying  and  outfitting  instruc- 
tional areas  (classrooms  and  laboratories)  with  fixtures  to 
accommodate  new  training  equipments. 

This  appears  to  be  the  case  in  transitions  from  conventional 
to  either  individualized  or  computer-based  instruction,  but  such 
costs  appear  to  be  modest  (see  Table  17).  The  Introduction  of 
individualized  or  computer-based  instruction  would  normally  re- 
quire replacement  of  traditional  classroom  desks  with  carrels 
and  might  require  the  extension  of  electric  service  to  individual 
student  positions  and  the  conversion  of  classrooms  to  larger 
learning  centers.  Introduction  of  computer-based  instruction 

TABLE  17.  COSTS  OF  FACILITIES:  LEARNING  CARRELS, 

ELECTRIC  AND  PNEUMATIC  LINES 


ITEM 

COST  PER  UNIT 

REFERENCE 

Individual  Learning  Carrel 

30  Carrels 

$90 

Dallman.  DeLeo,  etaL  (1977) 

20  Carrels 

260 

Steinkerchner,  Deignan,  etaL  (1977) 

Electric  and  Pneumatic 

Lines  (PLATO  IV) 

30  Carrels 

$61 

Dallman,  DeLeo,  etaL  (1977) 

20  Carrels 

141 

Steinkerchner,  Deignan,  etaL  (1977) 
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might  require  other  utility  service;  for  example,  PLATO  IV  termi- 
nals require  compressed  air  and  communication  lines. 

12 . Program  Management  and  Administration 

No  data  were  found  on  these  costs  of  instruction. 

1 3 . Student  Personnel:  Pay  and  Allowances  arc  Other  Costs 

Data  on  pay  and  allowances  are  widely  available.  Neverthe- 
less, such  data  appeared  only  in  three  studies  (see  Table  13). 

An  alternative  to  training  at  schools  is  to  conduct  the 
same  training  at  operational  sites.  It  has  been  proposed  that 
computer-based  instruction  would  increase  the  amount  of  opera- 
tional site  training  that  is  feasible  and  avoid  costs  of  relo- 
cating personnel  to  the  schools.  Relocation  costs  were  treated 
by  only  one  study.  Polcyn,  Baudhuin,  et  al.,  (1977)  present 
data  that  permit  estimation  of  transfer  costs  (including  per 
diem)  per  course:  $425  for  advanced  training  based  on  permanent 
change  of  station,  $400  for  advanced  training  based  on  temporary 
duty  transfer,  and  $140  for  initial  training  based  on  permanent 
change  of  station.  A significantly  higher  cost  ($825)  for  perma- 
nent change  of  station  for  advanced  training  is  cited  by  the  Air 
Force  in  "USAF  Cost  and  Planning  Factors"  (Air  Force  Regulation 
173-10) 

C.  DISCUSSION  AND  SUMMARY 

There  is  no  evidence  that  one  method  of  instruction  is  most 
cost-effective  for  all  types  of  military  training.  The  most 
cost-effective  method  for  a particular  situation  will  depend 
upon  such  factors  as  type  of  course  material,  location  of  instruc- 
tion, numbers  of  students,  and  life-span  of  the  training.  It  is 
apparent  that  the  cost  data  currently  available  make  it  impossible 
to  examine  satisfactorily  the  conditions  which  would  make  a par- 
ticular method  of  instruction  the  most  cost-effective  alterna- 
tive . 
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Unless  there  are  key  studies  that  have  been  overlooked,  both  the 
quantity  and  quality  of  current  data  on  training  costs  are  meager. 

The  training-management  and  data-reporting  systems  currently 
employed  by  the  Services  do  not  provide  Information  needed  for 
evaluating  the  cost-effectivenesB  of  alternative  methods  of  in- 
struction for  two  reasons.  First,  they  provide  information  only 
to  the  level  of  complete  training  courses,  while  analyses  of  the 
cost  of  methods  of  instruction  need  data  that  can  be  associated 
either  exclusively  or  predominantly  with  a single  method  of  in- 
struction within  a course.  Second,  training  courses  use  many 
resources  provided  by  school  management  and  organizations  at 
higher  echelons  ( i.e .,  "training  support").  The  use  of  such 
resources  may  differ  significantly  between  different  methods  of 
instruction.  However,  current  reporting  systems  veil  the  cost 
differences  by  allocating  support  on  arbitrary  bases,  such  as 
averaging  across  all  students  located  at  a training  facility. 

data  reporting  systems  that  would  provide  information  suit- 
able for  cost-effectiveness  analyses  of  Instructional  methods 
would  be  more  complex  and  expensive  than  current  systems  for  two 
reasons.  The  first  is  the  straightforward  multiplication  of  the 
number  of  training  activities  whose  costs  may  be  separately 
identified  and  compiled;  that  is,  each  course  has  many  segments. 
The  second  reason  lies  in  the  structure  of  military  training. 

If  the  costs  of  training  support  functions  are  to  be  attributed 
in  other  than  an  arbitrary  manner,  they  must  be  initially  re- 
corded in  a way  that  empirically  associates  the  support  provided 
with  the  individual  course  segments  receiving  the  support.  Con- 
sidering the  size  and  complexity  of  military  training  organiza- 
tions, this  is  a task  of  great  magnitude. 

Two  steps  are  necessary  to  lay  a foundation  for  building  a 
data  base  on  the  costs  of  training.  The  first  is  to  formulate 
hypotheses  regarding  causal  relationships  between  training  pro- 
gram characteristics  and  resource  requirements  in  order  to 
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identify  the  types  of  data  that  should  be  collected.  The  second 
step  is  to  formulate  and  evaluate  alternative  schemes  for  col- 
lecting the  data. 

Table  18  identifies,  for  hypothetical  purposes  only,  some 
determinants  of  costs  for  various  methods  of  instruction,  based 
on  our  review  of  the  literature,  analogies  to  weapon  system 
costs,  and  intuition.  The  determinants  shown  are  respresentative, 
rather  than  exhaustive,  and  probably  encompass  only  the  more 
obvious  factors.  At  that,  the  table  shows  the  extensive  range 
of  cost  and  non-cost  data  required  to  assess  training  costs  in 
a manner  suitable  for  analyses  of  the  cost-effectiveness  of 
methods  of  instruction. 

We  note  that  training  cost  data  could  be  collected  in  three 
fundamentally  different  ways:  universally  (continually  on  all 
training  activities),  by  a sampling  procedure,  or  on  an  ad  hoc 
basis.  We  have  discussed  the  nature  of  each  of  these  methods 
but  have  not  evaluated  them  or  the  costs  and  benefits  associated 
with  each  alternative.  The  question  of  how  best  to  collect  data 
vin  the  costs  of  alternative  methods  of  instruction  is  a central 
issue  requiring  further  and  thorough  study. 
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TABLE  18.  HYPOTHESIZED  DETERMINANTS  OF  COSTS 


8.1 


V.  DISCUSSION 


Computer-assisted  and  computer-managed  instruction  for 
military  training  have  been  evaluated  in  about  30  studies  con- 
ducted since  1968.  Most  of  these  were  experiments  of  limited 
duration.  In  a few  cases,  CAI  still  remains  in  use,  e.g ., 

PLATO  IV  for  training  medical  technicians  at  Sheppard  AFB , Texas, 
and  vehicle  repair  mechanics  at  Chanute  AFB,  Illinois,  and 
TICCIT  for  training  S-3A  tactical  coordinators  at  Naval  Air 
Station,  North  Island,  San  Diego,  California.  There  have  been 
fewer  evaluations  of  CMI  systems  but  most  of  these  systems  are 
still  operating  after  4 or  more  years,  e.g.,  the  Navy  CMI,  Air 
Force  AIS,  and  Army  CTS . A wide  variety  of  courses,  involving 
both  the  acquisition  of  knowledge  and  performance  skills,  were 
included  in  these  evaluations. 

A.  MAJOR  FINDINGS 

The  principal  findings  are  summarized  in  Table  19.  Computer- 
assisted  and  computer-managed  instruction  are  as  effective  as 
conventional  instruction  when  measured  by  student  achievement 
at  school,  but  a more  direct  and  relevant  measure  of  effective- 
ness is  the  performance  of  graduates  on  Jobs  in  operational 
units.  Correlations  between  performance  in  school  and  on  the 
Job,  though  thought  to  be  high,  have  not  been  demonstrated  either 
for  computer-based  or  conventional  instruction. 

Computer-based  instruction  typically  saves  30  percent  or 
more  of  the  time  students  need  to  complete  the  same  courses  given 
by  conventional  instruction.  The  amounts  of  time  saved  range 
widely,  but  research  has  not  addressed  the  factors  that  could 
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TABLE  19.  SUMMARY  OF  FINDINGS  ON  CAI  AND  CMI , COMPARED 
TO  CONVENTIONAL  INSTRUCTION 


Measure 

Finding 

(Compared  to  Conventional  Instruction} 

Comments 

CA! 

WEU 

Student  Achievement 

Same  or  more 

. 

Same 

Performance  measured  only  at  school. 
Relation  between  performance  at  school 
and  on  the  job  not  demonstrated. 

Observed  differences  not  of  practical 
importance. 

Course  Completion 

Time 

No.  of 
Comparisons 

40 

8 

CMI:  Most  time  savings  maintained 

or  increased  with  extended  use. 

Time  saved 
(Median) 

29% 

44% 

Range 

•31  to  89% 

12  to  69% 

No.  of 
Comparisons 

5 

7 

Computer-support  saves  little  time  beyond 
that  of  individualized  instruction. 

Time  saved 
Individual- 
ted  In- 
struction 

64% 

51% 

CAI 

69% 

CMI  51% 

Student  Attrition 

About  the  same 

Slight  increase 
may  occur 

CAI:  very  limited  data 

CMI:  possiole  decline  in  student  quality 

Student  Attitudes 

Favorable 

Favorable 

Instructor  Attitudes 

Unfavorable 

Unfavorable 

Very  limited  data. 

Little  attention  given  to  instructors. 

Cost 

Less,  due  to 

student  time  savings  ( 

Less,  due  to  stu 
ent  time  savings 

Data  limited  and  incomplete. 

Cost-effectiveness 

Not  known  because  cost  data  are  limited 
and  incomplete. 

2 7 79  2 4-10  79 
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account  for  these  variations.  Thus,  no  explanation  can  be  given 
at  present  for  the  different  amounts  of  time  savings  found  between 
CAI  and  CMI , between  various  types  of  CAI,  between  various  types 
of  CMI,  between  different  courses,  between  different  types  of 
instructional  strategies  ( e.g . , drill  and  practice,  tutorial, 
simulation,  student  pacing  algorithms,  types  of  remediation)  and 
the  like. 

It  is  widely  believed  that  transforming  a course  from  con- 
ventional to  individualized  (or  self-paced)  instruction  saves 
student  time.  Three  explanations  are  generally  offered  for  this 
effect : 

• Faster  students  are  not  held  back  by  rates  of  presenting 
material  in  conventional  instruction  set  to  permit  85  to 
90  percent  of  the  students  to  complete  the  course. 

• Course  materials  are  reviewed  and  irrelevant  materials 
tend  to  be  eliminated  when  courses  are  modified  in  format 
from  conventional  to  individualized  instruction. 

• Special  remedial  materials  can  be  provided  to  students 

on  the  ba3is  of  information  gained  by  frequent  diagnostic 
testing  of  their  progress  through  a structure  of  rela- 
tively brief  lessons. 

Computer-assisted  and  computer-managed  instruction  are,  of 
course,  forms  of  individualized  instruction.  Little  attention 
has  been  given  to  the  incremental  benefits,  if  any,  that  computer 
support  may  bring  to  individualized  instruction  (without  computer 
support).  Some  data  were  found  where  student  performance  could 
be  compared  on  the  same  courses  given  by  conventional,  individ- 
ualized, and  CAI  or  CMI  instruction.  Student  achievement  at 
school  was  about  the  same  with  each  method  of  instruction. 
Individualized  versions  of  five  courses  saved  6^  percent  of  the 
time  required  by  conventional  instruction;  the  CAI  version  saved 
an  additional  5 percent  or  a total  of  69  percent.  For  seven 
other  courses,  the  individualized  and  CMI  versions  each  saved  51 
percent  of  the  time  required  by  conventional  instruction. 
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Transforming  a course  from  conventional  to  individualized 
instruction  is  expected  to  save  student  time,  as  occurred  here. 
But,  it  is  also  clear  that  the  addition  of  computer  support  to 
the  individualized  versions  of  these  courses  does  not  further 
increase  to  any  appreciable  degree  the  amount  of  student  time 
saved.  The  extent  to  which  shortening  the  course  may  have  con- 
tributed to  saving  student  time  cannot  be  determined  from  these 
studies.  Since  the  same  Instructional  material,  both  in  content 
and  in  structure,  was  provided  in  the  individualized  and  com- 
puter-based versions  of  these  courses,  there  is  no  special  reason 
to  expect  that  the  addition  of  computer  support  should  produce 
any  incremental  time  savings.  However,  there  is  a substantive 
question  as  to  whether  the  incremental  cost  of  computer  support 
in  these  cases  produced  Incremental  benefits.  The  particular 
studies  from  which  these  data  were  taken  did  not  address  this  _ 
issue.  It  is  not  implied  here  that  computer  support  per  ee  does 
not  produce  benefits  equal  to  or  greater  than  its  cost.  Computer 
support  to  an  instructional  program  may  bring  certain  unique 
benefits  such  as  reducing  the  number  of  instructors  and  support 
personnel  needed  for  instruction,  reducing  the  costs  of  main- 
taining student  records,  and  reducing  the  costs  of  modifying 
and  updating  courses  because  of  an  ability  to  keep  detailed 
records  on  student  performance.  Whether  the  costs  of  adding 
computer  support  (CAI  or  CM1)  to  individualized  Instruction 
(without  computer  support)  produces  benefits  equal  to  or  greater 
than  these  costs  is  ar.  issue  that  clearly  needs  careful  explora- 
tion. 

B.  COST -EFFECTIVENESS  OF  CAI  AND  CMI 

There  have  been  only  a few  attempts  to  estimate  the  cost- 
effectiveness  of  CAI  and  CMI  and  these  are  based  on  incomplete 
analyses  of  the  costs  of  instruction.  Table  20  summarizes  the 
results  of  these  studies.  All  of  them  are  based  on  the  premise 
that  the  amount  of  student  training  time  saved  by  a method  of 
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Instruction  provides  major  cost  savings;  the  amounts  of  cost 
savings  are  estimated  by  computing  the  pay  and  allowances  of 
students  for  the  amounts  of  student  time  saved  in  training;  the 
resultant  amounts  should  more  properly  be  called  "cost  avoidance 
savings".  This  procedure  was  applied  to  time  savings  due  to 
PLATO  IV,  Navy  CMI,  and  AIS  and,  in  one  case,  to  revised  course 
materials  in  a course  given  by  conventional  instruction.  Four 
of  these  studies  consider  other  costs  in  addition  to  those 
avoided  by  student  time  savings,  such  as  for  preparing  course 
materials,  purchase  or  use  of  computers,  and  the  number  of  in- 
structors required  by  each  method  of  instruction  (Crawford, 

Hurlock  et  al.  , 1976;  U.S.  Army  Ordnance  Center  and  School,  1975; 
Dallman,  DeLeo  et  al.  , 1977;  and  the  AIS  Service  Test,  described 
in  an  Air  Force  briefing,  1978). 

The  dollar  amounts  of  such  "savings"  could  be  large,  depend- 
ing, of  course,  on  the  number  of  students  assumed  for  these  es- 
timates, e.g .,  about  $10  million  a year  for  about  50,000  students 
instructed  in  FY  1977  by  the  Navy  CMI  system  and  about  $3  million 
a year  for  about  5500  students  instructed  in  FY  1973  by  the  Air 
Force  AIS  system.  According  to  two  cost-effectiveness  evalua- 
tions that  have  been  reported,  the  PLATO  IV  system  Is  Judged  to 
be  not  as  cost-effective  as  Individualized  instruction.  These 
conclusions  are  based  on  incomplete  cost  data  in  two  small-scale 
tests  (535  students  in  four  courses  at  U.S.  Army  Ordnance  Center 
and  School,  Aberdeen  Proving  Ground,  Maryland,  1975;  1261  students 
in  four  courses  at  Chanute  AFB,  Illinois;  Dallman,  DeLeo  et  al.  , 

1977) .  The  Air  Force  AIS  was  found  to  be  cost-effective,  compared 
to  instructor  supported,  self-paced  instruction  in  one  course 
(Inventory  Management)  but  not  in  three  others;  the  computer  costs 
which  made  the  latter  courses  not  cost-effective  were  Judged  to 

be  small  in  comparison  to  other  school  costs  (AIS  Service  Test, 

1978) .  Since  all  of  these  findings  are  based  on  incomplete  cost 
data,  the  findings  cannot  be  generalized  or  even  taken  seriously. 
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Other  benefits,  beyond  those  of  saving  student  training 
time,  are  often  said  to  occur  with  CAI  and  CMI,  largely  because 
the  computer  can  compile  records  and  direct  the  attention  of 
instructors,  on  the  basis  of  various  algorithms.  The  following 
list  is  illustrative  rather  than  complete: 

• More  precise  data  for  improving  and  updating  course 
materials 

• Improved  control  over  equipment,  facilities,  and  materials 
for  instruction 

• Improved  allocation  of  resources  among  students 

• Improved  ability  to  accommodate  fluctuations  in  student 
loads 

• Increased  student : instructor  ratios,  as  well  as  the 
ability  to  use  some  instructors  with  less  advanced  quali- 
fications 

• Reduced  need  for  support  by  noninstructional  personnel 

• Reduced  time  of  students  on  base  waiting  for  courses  to 
start 

• Reduced  time  of  students  on  base  waiting  for  orders  after 
completing  courses 

• Improved  integration  of  records  of  students  at  school 
with  those  in  central,  computer-based  personnel  files 

• Improved  utilization  of  instructors. 

Many  of  these  benefits  may  occur  with  the  use  of  CAI  and 
CMI.  None  of  them  have  been  included  in  any  cost-effectiveness 
evaluation  known  to  us . Records  kept  at  Lowry  AFB  for  students 
instructed  by  the  AIS  show  that,  compared  to  prior  periods,  they 
spend  less  time  waiting  to  enter  a course  and  waiting  for  an 
assignment  after  completing  a course.  Records  kept  by  the  Navy 
CMI  system  show  that  the  average  on-board  count  of  students  in 
school  has  been  reduced  for  those  instructed  by  that  system: 
the  extent  to  which  this  may  be  attributed  to  various  benefits 
has  not  been  examined. 
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C.  WHAT  SHOULD  BE  DONE  NEXT? 


The  potential  value  of  computer-assisted  and  computer- 
managed  instruction  for  military  training  rests  primarily  on 
findings  that  (1)  computer-assisted  and  computer-managed  instruc- 
tion save  30  percent  or  more  of  the  time  (median  value)  required 
by  students  under  conventional  instruction  and  that  (2)  student 
achievement  at  school  is  about  the  same  with  computer-assisted 
and  computer-managed  instruction  as  with  conventional  instruc- 
tion. However,  these  results  do  not  necessarily  imply  that 
computer-assisted  and  computer-managed  instruction  are  cost- 
effective  because  of  fundamental  problems  with  the  measures  of 
effectiveness  and  of  cost  used  in  the  studies  from  which  these 
results  are  taken.  Effectiveness,  as  measured  by  student  achieve- 
ment at  school,  is  not  necessarily  a measure  of  performance  by 
course  graduates  in  relevant  jobs  after  they  leave  school.  Data 
on  the  costs  of  alternative  methods  of  instruction  reported  in 
various  studies  are  essentially  incomplete,  particularly  with 
respect  to  courseware,  student : instructor  ratios,  support  and 
management  services;  this  applies  both  to  computer-based  and 
conventional  instruction.  The  results  that  have  been  reported 
are  limited  to  obvious  costs  observed  during  experiments  ( e.g ., 
preparation  of  courseware,  rental  of  computers)  and  do  not  con- 
sider long-term  costs  associated  with  operational  applications 
(e.g.,  numbers  of  instructors  and  support  personnel,  revisions 
to  course  materials,  maintenance  of  software  and  facilities, 
management).  Next,  we  discuss  steps  that  should  be  taken  to 
remedy  these  deficiencies. 

1 . Measures  of  Effectiveness 

There  is  a need  to  compare  performance  on  the  job  of  stu- 
dents instructed  in  the  same  courses  by  alternative  methods  of 
instruction.  In  practice,  comparisons  will  be  required  between 
conventional,  individualized,  and  computer-assisted  or  computer- 
managed  Instruction.  The  general  absence  of  objective  data  on 
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the  performance  of  students  on  jobs  after  graduation  from  mili- 
tary training  courses  is  a major  deficiency  of  research  on  many 
aspects  of  military  training  and  is  not  limited  solely  to  deter- 
mining the  cost-effectiveness  of  computer-based  instruction. 

(See  McCluskey,  Trepagnler,  Cleary  et  al.  , 1975;  Pickering  and 
Anderson,  1976;  and  Foley,  197^,  1975,  for  recent  efforts  on 
measurement  of  Job  performance  in  the  Army,  Navy,  and  Air  Force, 
respectively.  Note:  "...  major  assessment  programs,  either 
within  or  outside  the  military,  that  rely  on  performance  tests 
as  their  primary  data  source  are  almost  non-existent."  Pickering 
and  Anderson,  1976,  p.  3<)  It  is  also  important  to  collect  on- 
the-job  performance  data  for  oeveral  time  intervals  after  stu- 
dents leave  school  ( e.g .,  3,  6,  and  12  months)  in  order  to  observe 
the  short  as  well  as  longer-time  effects  of  different  methods  of 
instruction.  It  may  turn  out  that  time  saved  at  school  must  be 
compensated  by  spending  more  time  in  training  on  the  Job  and  that 
deficiencies  In  performance  on  the  Job  attributed  to  one  method 
of  instruction  disappear  relatively  quickly.  Thus,  there  may 
well  be  a variety  of  trade-offs  between  the  costs  and  benefits 
of  various  methods  of  instruction  and  amounts  of  training  in 
schools  and  on  the  job. 

It  would  be  a major  undertaking  to  develop  objective  methods 
of  measuring  performance  on  the  Job  and  to  collect  on-the-job 
performance  data.  If  the  school-job  correlations  are  found  to 
be  high,  we  would  have  a basis  for  accepting  student  achievement 
at  school  as  a proxy  for  the  measurement  of  performance  on  the 
Job.  At  present,  we  do  not  know  the  extent  to  which  such  cor- 
relations may  exist  and,  if  they  do,  that  they  have  about  the 
same  magnitude  for  various  methods  of  instruction,  for  various 
types  of  courses,  and  for  varying  periods  of  time  on  the  Job 
after  leaving  school.  There  should  also  be  a feasibility  study 
to  examine  the  advantages,  disadvantages,  and  costs  of  various 
methods  of  collecting  and  reporting  on-the-job  performance  data. 
Although  these  data  are  proposed  here  to  evaluate  the 
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effectiveness  of  various  methods  of  training,  they  would  also  be 
useful  for  other  purposes  such  as  (1)  setting  standards  for 
recruitment  and  advancement  and  (2)  estimating  the  technical  and 
maintenance  readiness  of  the  operational  forces. 

2.  Measures  of  Cost 

Available  cost  data  are  unsatisfactory  for  conducting  cost- 
effectiveness  evaluations  of  computer-based  and  other  methods  of 
instruction  used  in  military  training.  Data  on  the  costs  of 
instruction  provided  by  recent  analytical  studies  of  military 
training  are  incomplete.  Further,  since  such  cost  data  come  from 
experiments  that  were  limited  with  respect  to  amounts  of  course- 
ware, numbers  of  students,  and  duration  of  the  experiments,  it 
is  questionable  whether  the  results  should  be  used  to  estimate 
the  costs  of  Instruction  under  operational  conditions.  Data 
collected  through  the  Services'  current  cost-reporting  systems 
are  not  satisfactory  because  costs  are  identified  only  with 
complete  training  courses  and  the  costs  of  training  support  are 
allocated  to  courses  on  arbitrary  bases  not  related  to  actual 
utilization  or  requirements  by  specific  courses.  Cost  element 
structures  for  collecting  such  data  are  identified  in  this  paper 
and  elsewhere.  (See  Petruschell  and  Carpenter,  1972;  Braby, 
Henry,  Parrish,  and  Swope,  1975;  Seidel  and  Wagner,  1977;  and 
McDonnell  Douglas  Astronautics  Company  East,  1977c). 

A comprehensive  effort  to  collect  data  on  all  the  relevant 
costs  on  all  methods  of  instruction  used  in  military  training 
would  be  a very  large  effort.  In  effect,  however,  identification 
of  the  major  cost  drivers  for  various  methods  of  instruction  is  a 
necessary  condition  for  assessing  the  cost-effectiveness  of 
feasible  alternatives.  Decisions  about  using  or  not  using  new 
methods  of  instruction  must  be  made  now  without  benefit  of 
reliable  cost  data.  However  desirable,  a large-scale  effort  to 
collect  such  data  cannot  be  recommended  now  without  further 
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examination  of  its  scope,  benefits,  and  costs.  Instead,  initial 
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efforts  should  be  directed  towards  evaluating  alternative  ways 
of  developing  a data  base.  In  addition,  cost  data  should  be 
collected  on  major  Instructional  systems  that  have  recently  come 
into  use  and  on  new  ones  being  considered  for  procurement.  These 
systems  are  identified  below: 

a . Air  Force  Advanced  Instructional  System  (AIS).  The 
current  AIS  incorporates  capabilities  for  research  that  would 
not  be  needed  in  an  operational  version.  Thus,  cost  data  are 
needed  on  an  operational  AIS  and  on  alternative  methods  of  in- 
struction for  technical  training.  The  recent  AIS  Service  Tesc 
(unpublished  as  of  February  1979)  suggested  that  AIS  was  cost- 
effective  compared  to  instructor-managed  instruction  only  in 
one  of  four  courses  used  in  that  evaluation. 

b . Navy  Computer  Managed  Instruction  System  (Navy  CMI). 
Published  information  suggests  that  the  Navy  CMI  system  saves 
student  time  and  thereby  avoids  costs.  However,  complete  cost 
data  on  this  system  have  not  been  published.  For  cost- 
effectiveness  evaluation,  cost  data  are  also  needed  for  compar- 
able courses  using  individualized  and  conventional  Instruction. 

c.  Navy  Aviation  Training  Support  System  (ATSS).  This 
planned  system  will  support  computer-managed  Instruction  for 
enlisted  men  and  officers  at  20  Naval  and  Marine  Corps  Air  Sta- 
tions; additional  units,  not  yet  planned,  could  support  Naval 
surface  warfare  facilities.  Cost  data  will  be  needed  to  support 
cost-effectiveness  evaluations  of  this  system  and  the  methods 

of  Instruction  that  it  would  replace. 

'd . Army  Automated  Instructional  Management  System  (AIMS). 
The  Army  AIMS  is  based  on  the  Navy  ATSS  and  the  above  remarks 
about  cost  data  also  apply  here.  About  20  units  will  be  ac- 
quired if  the  initial  installation  at  the  Field  Artillery  School, 
Fort  Sill,  Oklahoma,  is  found  to  be  effective. 

e . Marine  Corps  Communication-Electronics  School  CAI/CMI 
System.  The  Marine  Corps  plans  to  procure  a CAI/CMI  system  for 


the  Marine  Corps  Communication-Electronics  School,  Twenty-nine 
Palms,  California.  The  initial  configuration  of  this  system 
calls  for  installing  260  terminals  in  3 years  with  a potential 
growth  to  manage  2000  students  and  a maximum  of  1000  terminals. 

A preliminary  cost  analysis  is  being  conducted  by  the  Navy  Per- 
sonnel Research  and  Development  Center. 

3.  Research  and  Development 

An  "Integrated  DoD  Plan  for  R&D  on  Computers  in  Education 
and  Training"  was  prepared  in  draft  form  by  a tri-Servlce  group 
in  September  1975.  This  plan  should  be  revised  and  brought  up 
to  date  on  the  basis  of  more  recent  information  now  available 
on  computer-based  instruction.  Next,  consider  steps  that  should 
be  taken  to  Improve  our  ability  to  provide  more  relevant  and 
accurate  information  on  the  effectiveness  and  cost  of  various 
aspects  of  computer-based  Instruction. 

a . Factors  Which  Influence  the  Amount  of  Student  Time 
Saved . The  data  show  that  computer-assisted  and  computer-managed 
Instruction  save  appreciable  amounts  of  student  training  time 
compared  to  conventional  instruction.  However,  there  is  great 
variation  in  the  amounts  of  student  time  savings  found  in  many 
studies;  extreme  values  of  -31  to  89  percent  have  been  reported. 
Other  things  being  equal,  the  military  Services  should  obviously 
favor  those  applications  of  computer-based  instruction  which 
promise  greater  student  time  savi  .gs.  Thus,  research  is  clearly 
needed  to  explore  the  conditions  which  influence  the  amount  of 
student  time  saved.  Factors  which  could  influence  the  amount  of 
student  time  saved  by  CAI  and  CMI  probably  include  the  quality 
of  the  course  materials  (for  which  metrics  should  be  developed 
and  standards  set),  types  of  co;  ’ses  (i.e.,  some  may  save  more 
time  than  others),  and  Instructional  strategy  (e.g.  , effects  of 
combinations  of  drill  and  practice,  simulation  as  a method  of 
instruction,  frequency  of  testing,  length  and  difficulty  of  les- 
sons, and  methods  of  managing  students'  rates  of  progress  through 
a course).  On  a longer-term  basis,  it  is  important  to  know 
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whether  student  time  savings  accomplished  at  school  bring  any 
penalties,  such  as  in  quality  of  performance  on  the  Job  and  a 
need  for  additional  on-the-job  training. 

b.  Student  Attrition  with  CMI.  Currently  available  data 
suggest  that  student  attrition  with  CMI  may  be  somewhat  larger 
than  that  with  conventional  instruction.  However,  it  is  not 
clear  that  the  observed  increases  in  attrition  are  due  primarily 
to  computer-managed  instruction  oecause  the  qualifications  of 
students  in  these  courses  appeared  to  drop  at  the  same  time. 

Other  factors  may  also  be  Involved,  e.g .,  changes  in  the  number, 
quality,  and  support  provided  by  instructors.  Put  simply,  there 
is  a need  to  determine  the  extent  to  which  CMI  and  other  factors 
may  increase  attrition  of  students,  compared  to  the  rates  that 
occur  with  other  methods  of  instruction. 

c.  Role  of  Instructors  in  Computer-Based  Instruction.  The 
role  of  instructors  probably  differs  significantly  in  conven-  - 
tlonal,  individualized,  computer-assisted,  and  computer-managed 
instruction.  Yet  the  benefits  to  be  derived  from  each  method  of 
instruction  surely  requires  that  instructors  perform  adequately 
the  particular  functions  required  of  them  in  each  case.  Only  a 
few  studies  consider  the  attitudes  of  instructors  to  CAI  or  CMI: 
all  of  these  are  unfavorable  in  comparison  to  conventional  in- 
struction. The  limited  amount  of  data  cannot  be  regarded  as 
conclusive.  Thus,  there  is  a need  to  develop  more  reliable  in- 
formation on  the  attitudes  of  instructors  to  all  methods  of 
Instruction  used  at  present  by  the  military  Services.  The  survey 
instruments  should  be  diagnostic  in  nature  so  that  steps  could 

be  taken  later  to  remedy  problems  that  may  be  Identified,  e.g., 
experience  and  training  of  instructors,  relevance  of  this  train- 
ing to  their  jobs  as  instructors,  and  areas  where  instructors 
believe  that  problems  exist.  Emphasis  should  be  given  to 
(1)  computer-managed  instruction,  since  that  is  in  greater  use 
by  the  military  Services  than  is  computer-assisted  instruction 
and  to  (2)  individualized  instruction,  because  this  method  of 
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instruction  is  a reasonable  alternative  to  conventional  as  well 
as  to  computer-managed  instruction.  On  a longer-term  basis, 
steps  should  be  taken  to  identify  what  instructors  should  do  to 
make  computer-assisted  and  computer-managed  instruction  most 
effective  at  least  cost. 

d.  Comparison  of  Individualized  and  Computer-Based  Instruc- 
tion . A significant  finding  of  this  paper  is  that,  compared  to 
conventional  instruction,  computer-based  instruction  saves  no 
more  time  than  does  individualized  instruction.  Student  achieve- 
ment in  school  was  about  the  same  in  all  cases.  The  obvious 
question  is  whether  computer-assisted  and  computer-managed  in- 
struction provide  benefits,  greater  than  those  of  individualized 
instruction,  that  are  worth  their  incremental  costs.  On  the 
surface,  it  does  not  appear  that  the  additional  savings  in  stu- 
dent time  obtained  with  computer  support  would  be  sufficient  to 
pay  for  the  incremental  costs.  However,  this  observation  does 
not  consider  significant  cost  savings  that  computer-based  in- 
struction might  bring  in  a reduced  need  for  instructors,  improved 
record  keeping  and  management  of  students,  Instructional  materials, 
and  the  like.  An  analysis  of  the  costs  of  individualized, 
computer-assisted  and  computer-managed  instruction  for  the  same 
courses  Is  very  desirably. 

e . Methods  of  Collecting  Cost  Data  on  Methods  of  Instruc- 
tion. There  will  be  a continuing  need  to  collect  various  types 
of  cost  data  on  alternative  methods  of  instruction.  It  is  clear 
that  current  management  and  reporting  systems  do  not  provide  data 
that  are  satisfactory  for  use  in  analyses  of  methods  of  instruc- 
tion. Three  methods  of  collecting  cost  data  appear  to  be  avail- 
able : 

• Universal  collection 

• Sample  collection 

• Ad  hoc  collection. 
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A feasibility  study  is  needed  to  examine  the  advantages, 
disadvantages,  and  costs  of  various  ways  of  collecting  cost  data 
that  will  be  needed  to  support  cost-effectiveness  evaluations  of 
methods  of  instruction  in  the  near  future. 

f.  Other  Types  of  Research  and  Development.  Certain  types 
of  research  and  development  might  improve  the  effectiveness  and/ 
or  reduce  the  costs  of  computer-assisted  and  computer-managed 
instruction  in  military  training  but  there  Is  an  insufficient 
basis,  at  present,  to  recommend  funds  for  their  support.  Major 
items  of  this  type  would  probably  include  the  following: 

• Further  development  of  "Intelligent  computer-assisted 
instruction".  This  refers  to  the  use  of  the  computer 
to  model  each  student's  style  of  learning  and  to  use 
this  Information  to  construct  lessons  best  suited  to 
his  unique  needs  from  detailed  materials  stored  in  the 
computer.  In  effect,  this  eliminates  the  need  to  pre- 
pare complete  lessons  for  storage  in  the  computer,  as 

in  present  CAI  systems;  it  may  also  improve  the  effective- 
ness of  CAI  in  instructing  individual  students. 

• Improved  methods  of  preparing  courseware  to  reduce  high 
costs  currently  encountei’ed  in  this  area. 

• Improved  video  discs  and  solid-state  memories  to  signifi- 
cantly reduce  the  costs  of  major  components  in  computer- 
assisted  and  computer-managed  instruction,  particularly 
for  stand-alone  instructional  systems  used  away  from 
schools . 

• Improved  communications  for  computer-assisted  and 
computer-managed  instructional  systems  which  use  a large 
central  computer  to  support  many  terminals  at  different 
locations . 

• Investigate  the  feasibility  of  developing  a general  model 
for  use  in  cost-effectiveness  studies  of  military  training. 

• Determine  the  maximum  acceptable  costs  of  stand-alone 
terminals  or  other  system-design  concepts  for  them  to 
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be  cost-effective  in  selected  applications  of  CAI  and 
CMI,  i.e . , to  establish  cost  goals  for  the  improvement 
of  technologies  useful  for  computer-based  instruction, 
such  as  video  discs  and  solid-state  memories. 

• Define  ways  of  measuring  various  benefits  claimed  for 
CAI  and  CMI.  It  has  been  suggested  that  CAI  and  CMI 
provide  various  benefits  not  now  available  with  conven- 
tional instruction.  These  include,  for  example,  improved 
control  of  materials  and  facilities  required  for  instruc- 
tion, improved  utilization  and  assignment  of  instructors, 
more  accurate  information  derived  from  computer  records 
to  improve  lessons  and  tests,  more  accurate  and  complete 
student  records,  and  the  like.  No  data  have  been  offered 
to  support  such  claims  and  their  Impacts  are  not  clear 
on  the  costs  or  the  effectiveness  of  computer-based  in- 
struction. An  exploratory  study  would  be  useful  to 
define  ways  of  measuring  various  potential  benefits  of 
computer-based  instruction  and  of  identifying  the  cost 
and  other  data  that  would  be  needed  to  estimate  the 
magnitude  of  such  benefits. 

Given  the  absence  of  precise  information  on  the  major 
cost-drivers  in  computer-assisted  and  computer-managed  instruc- 
tion, for  either  large-scale  or  small-scale  installations,  it  is 
difficult  to  establish  any  amounts  or  priority  for  funding 
research  and  development  on  most  of  the  items  noted  above.  Nor 
is  it  clear  how  much  Improvement  is  needed  or  is  feasible  in  any 
of  these  areas  in  order  to  make  a significant  impact  on  the  cost 
and/or  effectiveness  of  computer-based  instructional  systems 
that  would  incorporate  such  improvements.  It  may  also  be  noted 
that  some  of  these  studies,  no  matter  how  desirable,  cannot  be 
undertaken  until  more  detailed  cost  data  become  available. 
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APPENDIX  A 


METHODS  OF  INSTRUCTION 


There  are  many  methods  of  instruction,  such  as  lecture,  dis- 
cussion, tutoring,  independent  study,  and  the  like;  20  different 
methods  of  teaching  listed  in  the  Rand  Corporation's  "Method  of 
Designing  Instructional  Alternatives  (MODIA)"  (Carpenter-Huf fman 
1977).  More  than  one  method  of  instruction  may  be  used  in  any 
course.  For  purposes  of  this  study,  we  group  methods  of  instruc- 
tion into  four  general  categories,  as  follows: 


Cateqory 

Examples 

Conventional  instruction 

Lecture,  discussion, 
demons  tratl on 

Individualized  instruction 

Programmed  instruction,  self- 
paced  Instruction,  preci- 
sion teaching 

Computer-managed  instruc- 
tion (CMI) 

Advanced  Instructional  Sys- 
tem ( A I S ) 

Navy  Computer  Managed  In- 
struction System  (Navy  CMI) 

Computer-assisted  instruc- 
tion (CAI) 

PLATO  IV,  TICCIT,  GETS,  LTS 

Tnese  categories  embody  several  key  distinctions:  conven- 
tional and  individualized  Instruction  do  not  require  computer 
support;  CAI  and  CMI  do,  and  are  generlcally  referred  t.  as  com- 
puter-based Instruction  (CBI).  Conventional  instruction  is 
aimed  (by  definition)  at  instructing  groups  of  students  at  the 
same  pace  (counselling  and  tutoring  of  individual  students  are, 
of  course,  attempts  to  individualize  instruction  within  the 
structure  of  conventional  Instruction);  the  other  three  methods 
are  designed  to  permit  each  student  to  learn  at  his  own  pace. 
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Each  of  these  methods  of  Instruction  is  described  below. 


A . 1 CONVENTIONAL  INSTRUCTION 

Conventional  instruction  typically  consists  of  lectures  and 
discussion  in  which  all  students  are  supposed  to  learn  the  same 
material  at  the  same  rate;  it  is  sometimes  referred  to  as  "lock- 
step"  instruction,  platform  instruction,  group  scheduling,  or 
block  scheduling.  A shortcoming  of  this  method  is  its  relative 
inflexibility,  particularly  with  large  groups  of  students. 
Students  differ  in  their  rate  of  learning  and  they  enter  a 
course  with  varying  degrees  of  knowledge  about  the  material  to 
be  taught.  However,  information  is  presented  at  a constant 
rate  for  all.  Even  if  the  rate  is  one  at  which  most  members 
of  the  class  can  learn,  it  will  necessarily  penalize  those  for 
whom  that  rate  is  either  too  fast  or  too  slow.  Thus,  slow 
learners  may  progressively  fall  further  behind  and  perhaps  fail 
and  fast  learners  may  simply  lose  interest  in  coming  to  class. 

All  students  spend  about  the  same  amount  of  time  in  the  class 
and  at  the  completion  of  the  course  differences  between  them 
are  reflected  in  how  much  has  been  learned,  i.e.,  in  final 
grades  from  norm-referenced  tests. 

A. 2 INDIVIDUALIZED  INSTRUCTION 

Individualized  instruction  is  a way  of  arranging  a curric- 
ulum into  small  lessons  and  tests  so  that  each  student  can  pro- 
ceed at  his  own  rate.  Although  the  term  "individualized  in- 
struction" could  also  apply  to  computer-based  instruction,  its 
use  here  will  be  limited  strictly  to  various  forms  of  Individual- 
ized instruction  conducted  without  a computer.  There  are  many 
types  of  individualized  Instruction  that  do  not  rely  on  com- 
puters such  as  self-paced  instruction,  programmed  instruction, 
personalized  system  of  instruction,  and  precision  teaching; 
these  differ  primarily  in  the  instructional  strategies  and  in 
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the  role  assigned  to  the  instructor  in  order  to  provide  each 
student  with  the  type  of  materials  best  suited  to  his  style  of 
learning. 

The  term  "self-paced"  needs  to  be  qualified.  Obviously, 
no  student  is  permitted  an  unlimited  amount  of  time  to  finish 
a course;  students  are  encouraged  to  request  assistance  from 
the  instructor  when  some  difficulty  is  encountered  or  the  in- 
structor may  intervene  when  he  observes  that  a student  is  pro- 
ceeding slowly.  Various  incentive  schemes  may  be  employed, 
such  as  assigning  a completion  date  to  each  student,  the  post- 
ing of  average  class  progress,  or  assistance  to  slower  students 
by  faster  ones.  Other  differences  among  methods  of  individual- 
ized instruction  concern  whether  all  students  must  progress 
through  the  same  set  of  lessons  ("straight-line")  or  whether  al 
ternative  lesson  materials  are  provided  ("branching"). 

Lessons  can  be  presented  in  booklets,  by  audio-visual  de- 
vices, in  a laboratory  set-up,  or  in  work  situations  such  as  a 
maintenance  shop;  the  lesson  material  can  consist  of  knowledge 
or  skill  or  both,  as  in  the  maintenance,  calibration,  and  re- 
pair of  equipment.  Instruction  is  oriented  to  the  complete 
mastery  of  lessons;  lessons  and  tests  are  tied  to  each  other. 

If  a student  passes  a test,  he  goes  on  to  the  next  lesson;  if 

not,  he  repeats  the  lesson  in  the  same  or  modified  form.  The 

» 

student  can  take  tests  without  taking  lessons  since  successful 
completion  of  a test  is  presumed  to  show  that  he  knows  the  re- 
quired material;  he  need  take  only  the  lessons  prescribed  for 
the  tests  that  he  failed.  It  is  not  a trivial  matter  to  deter- 
mine the  proper  amount  of  information  to  be  included  in  a sin- 
gle lesson,  to  decide  whether  a slide  or  a written  text  is  the 
more  effective  v/ay  of  presenting  certain  information,  and  to 
arrange  a sequence  of  lessons  that  is  efficient  for  instruction 
al  purposes.  The  essence  of  individualized  Instruction  is  to 
provide  a structure  of  lessons  and  tests  that  a student  can 


take  at  his  own  rate  to  master  an  entire  course.  It  does  not 
require  the  use  of  a computer. 

To  graduate,  all  students  must  pass  the  required  lesson 
and  end-of-course  tests.  The  instructional  material  is  pre- 
pared so  that  there  is  a high  probability  that  all  students 
can  complete  the  course.  Students  either  know  what  they  are 
supposed  to  know  or  they  do  not  graduate,  as  determined  by  cri- 
terion-referenced rather  than  norm-referenced  tests.  Differ- 
ences among  students  are  reflected  primarJ ly  in  the  amount  of 
time  they  need  to  complete  the  course,  although  grades  and  ^est 
scores  may  also  be  recorded. 

A. 3 COMPUTER-MANAGED  INSTRUCTION  (CMI) 

Individualized  instruction  permits  each  student  to  proceed 
at  his  own  pace,  but  it  generally  increases  the  Instructor's 
clerical  and  advisory  loads.  The  Instructor  needs  to  give  and 
score  more  tests  than  he  would  in  a conventional  classroom. 

He  must  keep  track  of  each  student's  rate  of  progress  on  almost 
a dally  basis  and  work  closely  with  students  when  they  have 
problems  with  any  part  of  the  course.  He  must  find  the  right 
lesson,  audio-visual  cassette,  or  test  bench  whenever  a student 
needs  it  and  keep  track  of  all  available  resources.  All  of 
these  functions,  and  certainly  all  of  the  clerical  and  bookkeep 
ing  tasks,  can  be  accomplished  readily  by  a modern  computer  (se 
Baker  1978  for  a recent  review). 

In  computer-managed  instruction  (CMI),  all  instruction 
takes  place  off-line,  i.e.y  away  from  the  computer.  The  point 
of  contact  between  the  student  and  the  computer  is  the  test 
which  accompanies  each  lesson.  The  test  may  be  given  either  on 
line  or  off-line  with  a machine-scorable  answer  sheet.  In 
either  case,  the  computer  scores  the  test,  immediately  reports 
the  results  to  the  student,  tells  him  which  lesson  to  take  next 
and  where  it  may  be  found  in  the  learning  center.  The  next 
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lesson,  of  course,  may  represent  progress  or  remediation;  if 
there  is  too  much  of  the  latter,  the  computer  advises  the  stu- 
dent to  consult  the  instructor. 

A CMI  facility  can  provide  many  of  the  services  associated 
with  training  such  as  the  following: 

• Testing 

-Scoring 

-Diagnosis 

-Prescription,  such  as  drill  and  practice,  or  remedia- 
tion (repetition  of  all  or  part  of  a lesson,  or 
assignment  of  the  student  to  parallel  tracks  that 
differ  in  the  level  of  difficulty  between  steps  or 
in  the  method  of  presenting  information). 

• Management 

-Monitoring 

-Pacing  students  according  to  predetermined  rules 

• Scheduling 

-Assignment  of  students 

-Optimum  allocation  of  instructional  materials  and 
facilities 

-Assignment  of  instructors 

• Administrative  record  keeping 

-Student  personnel  records 

-Student  test  data,  graduation 

-Instructor  records 

-Inventory  and  control  of  learning  resource  materials 
(films,  tapes,  workbooks,  projectors,  etc.) 

-Frequency  and  time  of  use  of  materials  and  facilities 

• Course  development 

-Since  test  results  show  how  students  answered  each 
test  item.  Information  is  available  to  identify 
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the  difficulty  of  the  different  parts  of  each  lesson. 
This  provides  a basis  for  modifying  the  course  and 
testing  the  effectiveness  of  various  improvements. 


Examples  of  CMI  systems  are: 

Army : Automated  Instructional  Management  System  (AIMS) 

Field  Artillery  School,  Fort  Sill,  Oklahoma 

Navy : Computer  Managed  Instructional  System 

Naval  Air  Technical  Training  Center, 

Millington,  Tennessee. 

Aviation  Training  Support  System  (ATSS) 

[planned  for  about  20  Naval  and  Marine  Corps 
Air  Stations;  previously  called  Versatile 
Training  System  (VTS)]. 

Air  Force:  Advanced  Instructional  System  (AIS) 

Air  Force  Technical  Training  Center 
Lowry  Air  Force  Base,  Denver,  Colorado 

A. 4 COMPUTER-ASSISTED  INSTRUCTION  (CAI) 

In  Computer-Assisted  Instruction  (CAI),  the  instructional 
materials  are  stored  in  and  presented  by  the  computer.  Each 
student  Interacts  individually  with  segments  of  the  material 
and  takes  tests  through  some  type  of  terminal.  The  typical  ter- 
minal contains  one  or  more  output  and  input  devices;  for  out- 
put, a television  or  plasma  panel  display,  screen  for  slides  or 
flche,  paper  printer,  loudspeaker;  for  input,  a keyboard,  tele- 
typewriter, pointing-type  capability,  graphic  tablet,  microphone. 

CAI  systems  generally  Include  many  terminals.  In  the  TICCIT 
(Time-shared,  Interactive,  Computer-Controlled,  Information  Tele- 
vision) system,  up  to  128  terminals  and  one  control  compute;  are 
located  at  the  same  site;  In  the  PLATO  IV  system  (Programmed 
Logio  for  Automatic  Teaching  Operation),  about  1000  terminals 
in  different  locations  are  linked  to  a central  computer  by  long- 
distance communication  lines.  In  principle,  each  CAI  terminal 
could  have  its  own  mini-  or  micro-computer,  but  such  an  arrrange- 
ment  is  expensive  at  present.  Systems  of  this  type  (called 
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"stand-alone")  have  been  developed  for  the  military  Services, 
e.g . , the  Lincoln  Terminal  System  (LTS)  and  the  General  Electric 
Training  System  (GETS). 

A narrow  definition  of  CAI  would  consider  it  to  be  a way 
("medium")  of  presenting  instruction,  comparable  to  TV,  slide, 
or  filmstrip  and  regard  its  flexibility  as  a form  of  CMI . How- 
ever, this  distinction  has  little  practical  meaning.  The  inter- 
active nature  and  flexibility  of  a CAI  system  distinguish  it 
from  CMI  or  programmed  instruction  of  any  variety.  With  appro- 
priate programming , CAI  permits  a dialogue  to  occur  between 
student  and  computer  on  every  frame  of  an  instructional  se- 
quence. The  computer  can  tutor,  prompt,  drill,  and  test  the 
student  on  a frame-by-frame  basis;  by  means  of  simulation,  it 
can  guide  and  test  the  student  on  complex  dynamic  processes. 

By  diagnosing  the  student's  progress,  it  can  identify  and  select 
the  material  best  needed  to  meet  specific  deficiencies;  it  could 
compose  (in  fact  create)  appropriate  lessons  and  tests  from  a 
large  store  of  elemental  materials. 

Although  CAI  is  defined  narrowly  as  a medium  of  presenta- 
tion, most  CAI  systems  also  provide  the  administrative  and  re- 
cord-keeping capabilities  inherent  in  CMI  systems. 

A developing  application  of  computers  to  instruction  is 
"Intelligent  Computer-Assisted  Instruction"  (ICAI).  In  ICAI, 
subject  matter  knowledge  is  stored  in  the  computer,  but  not  in 
the  form  of  previously  defined  lessons.  Instead,  the  computer 
models  each  student  and  selects  stored  subject  niktter  material 

to  construct  instructional  interactions  for,  individual  students. 

\ 

These  Interactions  can  be  very  flexible,  and  hold  some  promise 
for  "human-like"  tutoring  of  students. 

ICAI  is  a very  new  area  with  work  currently  underway,  spon- 
sored by  DARPA,  ONR,  and  others.  However,  no  data  are  available 
yet  on  costs  or  effectiveness,  and  therefore,  ICAI  will  not  be 
treated  further  in  this  paper.  There  can  be  little  question 
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about  the  superior  instructional  capabilities  provided  by  a 
CAI  system.  The  real  issue  is  whether  these  capabilities  are 
worth  their  cost. 

Examples  of  some  CAI  systems  evaluated  in  military  training 
are  listed  below.  These  are  described  in  Appendix  B. 

CTS  Computerized  Training  System 

U.S.  Army  Signal  School 
Fort  Gordon,  Georgia 

PLATO  IV  Programmed  Logic  for  Automatic  Teaching 

Operation 

Computer-based  Education  Research  Laboratory 
University  of  Illinois 

TICCIT  Time-shared,  Interactive,  Computer-Controlled, 

Information  Television 
The  MITRE  Corporation 

LTS  Lincoln  Terminal  System 

Lincoln  Laboratory 

Massachusetts  Institute  of  Technology 
Lexington,  Massachusetts 

GETS  General  Electric  Training  System 

General  Electric  Ordnance  Systems 

Electronics  Systems  Division 
Pittsfield,  Massachusetts 


The  idea  of  individualized  instruction,  which  underlies 
all  forms  of  computer-based  instruction,  long  precedes  the  dev- 
elopment of  modern  computers.  It  was  promoted  by  John  Dewey 
about  1900  at  the  University  of  Chicago  {and  by  Socrates  cen- 
turies ago).  Sidney  L.  Pressey  (1926,  1927)  designed  several 
mechanical  "teaching  machines"  which  provided  preprogrammed 
drill  and  practice  frames  as  well  as  automatic  self-scoring  of 
tests.  His  concept  was  to  provide  immediate  feedback  and  self- 
pacing in  education.  Testing  one  of  these  machines.  Little 
(193*0  found  that 


Students  immediately  apprised  of  their  test 
results,  and  given  opportunity  to  correct  deficiencies 
by  make-up  tests,  profit  markedly  in  terms  of  final 
examination  results  over  students  who  do  not  have  such 
advantage . 

Students.  . .given  opportunity  to  correct  de- 
ficiencies by  drill  and  by  make-up  tests,  likewise 
so  profit. 

The  greatest  benefit  accrues  to  students  who  usually 
score  in  the  lower  half  of  the  distribution,  although 
the  entire  group  moved  upward. 

(Reprinted  in  Lumsdalne  and 
Glaser,  I960,  p.  65) 

Pressey  expected  his  machines  to  produce  a "coming  indus- 
trial revolution  in  education"  but  this  did  not  occur.  In  1932, 
he  wrote,  "The  writer  has  found  from  bitter  experience  that  one 
person  alone  can  accomplish  relatively  little  and  he  is  regret- 
fully dropping  further  work  on  these  problems."  (Quoted  in 
Skinner,  1958),  Skinner's  teaching  machines  required  the  stu- 
dent to  compose  his  response  rather  than,  as  did  Pressey 's, 
select  it  from  a set  of  alternatives.  Skinner  presented  mate- 
rial in  a progression  of  small  steps,  each  of  which  the  student 
could  probably  understand  and,  in  so  doing,  become  ready  for  the 
next;  the  student  got  feedback  by  being  reinforced  for  every 
correct  response.  (Skinner,  195^,  1958).  Skinner's  work  was 
done  at  the  right  time  and  had  impact.  Sy  1962,  over  80  differ- 
ent teaching  machines  and  630  instructional  programs  were  com- 
mercially available;  six  machines  were  computer  controlled 
(Aeronutronics , DEC,  Marquardt,  Rheem,  TRW,  and  USI  Robodyne ; 
Finn  and  Perrin,  1962).  IBM  simulated  a Skinner  teaching 
machine  on  an  IBM  650,  starting  in  1958,  and  work  on  PLATO 
started  at  the  University  of  Illinois  in  i960.  (Rath,  Anderson 
and  Brainerd,  1959;  Alpert  and  Bitzer,  1970).  Some  noted  engi- 
neers contributed  to  this  progress.  In  a well-known  article, 
Vannevar  Bush  (19^5)  predicted  a computerized  desk  which  would 
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contain  a large  volume  of  library  materials  and  monitor  a 
reader's  progress.  In  1S57,  Simon  Ramo  described  how  "teaching 
engineers"  with  pushbutton  classes  and  memorizing  machines  could 
help  meet  the  increasing  need  for  more  education  in  a growing 
technical  society.  The  Department  of  Defense  (AFOSR,  APPTRC, 
NTDC,  ONR)  supported  much  of  the  original  R&D  on  teaching 
machines  during  the  1950s. 

A. 5 ADVANTAGES  AND  DISADVANTAGES  OF  VARIOUS  METHODS 

OF  INSTRUCTION 

The  Services  conduct  individual  training  at  126  different 
facilities  076  for  specialized  skill  training  alone);  each 
Service  offers  3000  to  AOOO  courses.  The  average  number  of 
students  on  board  in  these  courses  can  vary  at  any  time  from 
about  10  to  2,500  (the  largest  is  for  a course  in  propulsion 
engineering  at  the  Navy  Training  Center,  Great  Lakes,  Illinois). 
Course  length  can  vary  from  days  to  months.  It  is  not  likely 
that  any  method  of  instruction  is  best  suited  for  all  courses 
in  this  wide  spectrum  of  requirements.  With  this  obvious  quali- 
fication in  mind,  it  is  useful  to  compare  the  advantages  and 
disadvantages  generally  attributed  to  the  four  methods  of  in- 
struction that  have  been  described.  The  main  points  are  summa- 
rized in  Table  A-l. 

A. 5.1  Conventional  Instruction 

Conventional  instruction  permits  flexibility  in  presenta- 
tion of  material  to  suit  the  needs  of  Individual  students,  pro- 
vided the  instructor  is  free  to  do  so.  Human  contact  can  serve 
to  motivate  students.  The  standard  rate  of  progress  is  estab- 
lished to  produce  some  goal  established  by  policy,  e.g.,  that 
at  least  90  percent,  of  the  students  master  the  course;  the  slow 
learners  may  fall  too  far  behind  to  catch  up;  the  fast  learners 
waste  time  and  may  lose  interest,  individualized  attention 
becomes  increasingly  difficult  as  class  size  increases. 
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TAGLE  A-l.  ADVANTAGES  AND  DISADVANTAGES  OF  FOUR 
METHODS  OF  INSTRUCTION 


*T*00  OF 

INSTRUCTION ADVANTAGES DISADVANTAGES 


Convention*! 

Instruction 


Low  delivery  cost  for  Urge  class  sizes. 

Flexible  ir*  mode  of  Instruction,  use  of  media,  course 
content,  emphasis. 

Direct  human  contact. 

Simplifies  planring  because  *11  students  must  pro- 
gress «t  the  same  rate  and  complete  the  course  at 
the  same  time. 

Instructors  prepare  Instructional  material. 

Grades  at  end  Of  course  define  Now  much  each  student 
his  learned  In  relation  tc  a normative  sample. 


Not  all  students  are  able  to  progress  at  same  rate 
with  equal  mastery. 

Low  student: instructor  ratios  increases  costs. 

Individual  attention  to  students  difficult  as  class 
size  Increases. 

Differences  between  Instructors  may  lead  to  non- 
uni form  achievement. 

Fast  learners  may  lose  interest  in  course. 

Slow  learners  become  Increasingly  penalized. 

Load  on  Instructor  'or  scoring  Tests  and  managing 
students'  progress  Increases  markedly  with  class 
size. 

Difficult  to  Insure  student  mastery  of  training  ob- 
jectives since  norm* referenced  rather  chan  objective- 
referenced  testing  procedures  are  used. 

Difficult  to  Insure  that  instructors  present  the 
relevant  Instructional  Information . 


Individual Ized 
Instruction 


Explicit  course  and  lessor  objectives. 

Standardized  Instruction. 

All  students  progress  at  their  own  rate  (i.e.,  slow 
learners  do  not  hold  up  faster  learne-s). 

Students  can  skip  course  material  they  already  know, 
as  shown  by  ©reassessment  tests. 

Testing  and  evaluation  closely  tied  to  small  lesson- 
steps. 

Lessons  generally  one  track. 

All  graduates  are  warranted  to  know  the  required 
Information  (i.e..  Students  pass  the  required 
tests  Or  they  do  not  graduate). 

Instructors  can  concentrate  their  t4me  on  those 
students  who  need  assistance  at  both  ends  of  the 
distribution. 

Permits  use  of  1n$tr«  tor's  aides,  thereby  reducing 
average  level  of  qualification  required  of  instruc- 
tors. 

Permits  wide  use  of  different  Instructional  media. 

Instructors  relieved  from  rote  repetition  of  basic 
materials. 

Instructors  can  h*ve  time  to  address  concepts  as 
well  as  student  evaluation,  motivation,  and 
enrichment . 


Nigh  Initial  costs  for  development  of  course  materials, 
carrels,  audio-visual  equipment,  etc. 

Increases  demand  for  qualified  personnel  to  prepare 
Instructional  materials. 

Requires  changes  In  the  Instructor's  role  In  conven- 
tional instruction. 

Load  on  Instructor  for  scoring  tests  and  managing 
students’  progress  increases  markedly  with  class 
size. 


CM  | 


All  of  those  for  individualized  instruction,  plus: 

Reduces  demand  for  number  of  Instructors. 

Presentation  of  lessons  and  taking  of  tests  not 
dependent  on  computer. 

Automated  test  scoring,  evaluation,  prescription . 

Student  progress  monitoring. 

Multi-track  lessons  readMv  handled. 

Automated  student  management,  record-keeping 
and  schedul Ing. 

Resource  ma^gefnent. 

Detailed  information  routinely  available  for 
evaluating  and  modifying  lessons  and  tests. 

Manual  scoring  possible  if  computer  and/or 
codihunlcltlon  falls. 

Predict  graduation  date,  based  on  rate  of  student 
progress  In  course  and  personal  data  In 
student's  file. 

Provides  data  base  for  research,  course  develop- 
ment, and  management  decisions. 


All  of  those  for  indl vldual ized  Instruction,  plus: 

High  Initial  costs  for  courseware,  CPU,  terminals 

High  >perat1ng  costs  for  conwunlc atlons , where 
required. 

Instructional  material  poorly  matched  to  students’ 
abilities  and  expectations  may  discourage  students 
and  reduce  effectiveness. 

Instructor's  attitudes  often  unfavorable. 

Scoring  and  student  management  Inoperative  if  com- 
puter and/or  communications  fall. 


(contl nu*d  i 
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TABLE  A-l.  (Continued) 


AO  V A N 1AG£S 0ISAQVAST  kGU 

CMl  A wide  array  of  courses  can  t>e  offered  with  few 

instructors  (or,  *t  remote  fecilities,  without 
Instructors,  limited  primarily  by  cost  factors 
and  availability  of  communications) • 


CAI 


All  yf  those  fo"  CHI  --  except  that  ability  to 
operate  when  computer  i*  Inoperative  Is  extremely 
limited  --  plus: 

Very  flexible  ..mans  for  presenting  material  end 
taking  tests  via  computer. 

Interactive  tutorial  modes  feasible, 

Simulation  of  processes  end  equipment  feasible. 

Computational  elds  readily  available. 

Can  provide  detailed  information  needed  to  im- 
prove specific  lessons  and  tests, 
student  success  with  various  subjects,  method 
of  presentation  (graphic,  text).  Instructional 
strategy,  delay  times. 

Can  provide  instructors  with  data  bases,  formats, 
guidelines  for  developing  improved  course  ia- 
terlels 

Facilitates  maintaining  security  of  tests. 

Proba.  ly  the  greatest  degree  of  tndlvldu* i ued 
Instruction  currently  available,  except  where 
very  low  student: Instructor  ratios  are  accept- 
able. 


All  of  those  for  CMl , plus: 

Instruction  becomes  difficult  when  computer  re- 
sponses are  detayed. 

No  Instruction  possible  when  computer  and/or 
conmunl cat  Ions  fall. 
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Instructors  favor  this  perhaps  because  the  role  offers  high 
visibility  and  a position  of  authority.  Low  investment  costs 
are  required  for  current  courses  because,  generally  speaking, 
th^y  were  paid  long  ago  when  courses  were  originally  developed. 

A teacher  can  reach  a large  audience  in  a lecture  hall  and 
multiply  this  reach  almost  without  limit  by  means  of  a movie  or 
TV  recording.  However,  as  KcKeachie  (1970,  p.13)  said  in  a 
frequently  quoted  comment: 

The  technological  bottleneck  in  education 
is  that  we  have  no  device  that  allows  a teacher 
to  listen  or  respond  to  more  than  one  student 
at  a . ' me . 

A . 5 . 2 Indi vidual 1 zed  Instruction 

Individualized  Instruction,  whether  delivered  by  programmed 
texts,  a multi-media  approach,  or  by  computer,  has  the  advantage 
of  dealing  more  efficiently  with  different  rates  of  learning  and 
different  amounts  of  prior  knowledge  among  students.  Its  major 
limitations  lie  in  the  efforts  required  to 

• Identify  the  specific  lesson  objectives  which,  In 
some  progression,  satisfy  the  overall  course 

obj  ectives 

• Develop  instructional  material  that  delivers  the 
required  information 

• Develop  tests  that  measure  the  student’s  progress  on 

each  lesson  and  diagnose  the  types  of  remediation 

that  may  be  required,  and 
s 

• Prepare  the  remedial  treatments. 

Course  materials  must  be  pretested  with  students  to  ensure  that 
lessons  are  neither  too  easy  nor  too  difficult  for  students;  it 
is  often  necessary  to  modify  lessons  in  order  to  "validate" 
them.  Developing  courses  for  individualized  instruction  requires 
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qualified  subject  matter  experts  and  curriculum  development 
experts  who  are  not  always  available  when  needed. 

Individualized  instruction  inherently  makes  more  efficient 
use  of  the  student's  time  than  does  conventional  classroom  in- 
struction since  each  student  can  start  a course  when  he  arrives 
at  a school  without  having  to  wait  for  a class  of  some  optimum 
size  to  be  formed.  Each  student  can  graduate  when  he  has 
mastered  the  specified  material.  Because  each  lesson  has  a 
test,  the  instructor  can  identify  students  who  are  falling  be- 
hind early  in  the  course,  give  them  personal  guidance,  and  pro- 
vide them  with  material  selected  to  deal  with  their  particular 
problems.  This  contrasts  with  conventional  instruction  where 
tests  tend  to  measure  student  competence  but  do  not  provide  a 
basis  for  remediation;  te.%s  are  also  given  less  frequently. 

In  this  environment,  an  instructor  must  grade  tests  more 
often  than  in  a conventional  classroom;  this  in  Itself  can  be- 
come a large  load.  Since  the  instructor  has  more  detailed 
knowledge  about  each  student's  progress,  he  can  provide  more 
individualized  guidance  than  is  possible  in  a conventional 
setting. 

A . 5 . 3 Computer-Managed  Instruction 

CMI  provides  a means  to  handle  many  of  the  administrative 
loads  encountered  in  individualized  instruction,  such  as  scoring 
and  prescribing  lessons,  Identifying  students  who  need  remedia- 
tion, managing  and  scheduling  instructional  resources,  and  pre- 
dicting course  completion  times  so  that  students  can  be  sent 
promptly  to  their  next  assignment.  CMI  systems  readily  comple- 
ment and  can  be  tied  into  automated  manpower  and  personnel 
management  systems  that  are  used  by  all  military  Services. 

A < 5 . 4 Computer-Assisted  Instruction 

All  of  this  can  also  be  provided  by  CAI.  The  preparation 
of  course  material  for  presentation  by  CAI  is  similar  in  concept 
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but  often  more  sophisticated  than  that  for  programmed  instruc- 
tion or  for  CMI . It  is  obvious  that  it  is  probably  inefficient 
to  use  computers  to  present  programmed  materials  ( i.e to  "turn 
pages")  unless,  of  course,  it  costs  less  to  do  so  by  computer 
than  by  printing  the  same  material  on  paper  (which  may  soon  be 

the  case  if  frequent  reprinting  is  required  to  update  instruc- 

tional material  and  if  the  costs  of  word-processing  types  of 
computer  systems  continue  to  be  reduced).  The  major  advantage 
of  CAI  is  that  it  permits  extraordinary  flexibility  in  querying 
and  prompting  each  student,  a process  which  permits  the  computer 
to  select  material  of  a complexity  or  level  of  difficulty  most 

likely  to  meet  each  student's  rate  of  learning  and  best  suited 

to  deal  with  his  misunderstandings  and  errors.  Such  a dialogue 
is  highly  motivating  and  serves  to  engage  the  attention  of  the 
student.  It  is  Indeed  possible  that  a student  may  get  more 
Individual  attention  from  a computer  than  he  may  get  from  many 
human  instructors,  particularly  where  large  classes  are  involved. 
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TYPES  OF  COMPUTER-ASSISTED  AND  COMPUTE R- MANAGE D 
INSTRUCTION  SYSTEMS 

Almost  every  existing  computer-based  instructional  system, 
whether  CAI  or  CMI,  is  tailored  to  some  particular  requirement 
such  as  the  number  and  types  of  courses,  location  of  school(s), 
student  flow,  and  the  availability  of  instructors,  resources, 
and  funds.  Hardly  less  important  is  the  influence  of  such  fac- 
tors as  whether  the  method  of  instruction  should  be  primarily 
CMI  or  CAI,  and  whether  the  required  computer,  wherever  it  may 
be  located,  will  be  used  solely  for  instructional  purposes  or 
also  for  some  nonlnstructlonal  purposes,  such  as  maintaining 
medical  and  personnel  records,  base  accounting,  preparing  pay- 
rolls, and  the  like.  For  such  reasons,  there  are  no  "standard" 
computer-based  instructional  systems  and  few  are  likely  to  be 
identical. 

An  inventory  of  computer-based  instructional  systems  in 
current  use  by  the  military  services  does  not  exist.  In  a 197^ 
survey,  it  was  found  that  computer-based  instruction  was  used  by 
the  Army  In  217  courses,  the  Navy  and  Marine  Corps  in  102,  and 
the  Air  Force  in  210.  Compared  to  all  other  military  instruc- 
tion, this  accounted  for  2,  3.6,  and  2 percent,  respectively, 
of  all  courses  (Sherron,  1976).  In  another  survey  of  116  Army 
courses  at  16  schools,  also  in  197^,  use  of  the  computer  in  these 
courses  varied  over  the  widest  possible  range,  i.e.,  from  0.01 
to  100  percent  (Rich  and  VanPelt,  197^). 


B-3 


1 


The  military  Services  have  conducted  experiments  on  most 
of  the  CAI  and  CMI  systems  that  have  been  available  since  about 
1965.  Even  the  most  recent  acquisitions,  such  as  the  Air  Force 
Advanced  Instructional  System  or  the  Navy  Computer  Managed  In- 
struction system,'  represent  computer  technology  of  the  early 
1970s.  Only  the  major  features  of  the  systems  used  in  military 
studies  are  described  here  (see  Sherron  1975  for  additional 
Information) . 

B . 1 ARMY  COMPUTERIZED  TRAINING  SYSTEM 
* . 

The  Army  Installed  a prototype  Computerized  Training  System 
(CTS)  at  the  U.S.  Army  Signal  School,  Fort  Gordon,  Georgia,  over 
the  period  197^-1976.  The  program  to  evaluate  CTS  for  Army  use 
was  called  Project  Abacus,  a name  used  interchangeably  with  CTS. 
The  Army  refers  to  CTS  as  a CAI/CMI  instructional  system.  There 
are  128  terminals  in  CTS,  each  with  a Visual  Display  Unit  and  a 
keyboard.  CTS  also  contains  six  mini-computers  (PDP-11/351; 
four  of  these  computers,  called  Display  Controllers,  support  32 
CAI  terminals  each;  the  two  other  computers  serve  as  System 
Controller  and  Data  Base  Controller,  respectively. 

The  CTS  features  a fast  response  time:  each  of  the  32 
terminals  in  a cluster  can  be  updated  in  less  than  250  milli- 
seconds. Three  courses  were  developed  for  CTS: 

• Field  Repair  Radio  Course  (31E20) 

• Teletypewriter  Equipment  Repair  Course  (31J20) 

• Avionics  Communications  Equipment  Repair  Course  (35L20). 

CTS  was  applied  to  these  courses  at  Fort  Gordon  after 
feasibility  and  follow-up  studies  (conducted  at  the  U.S.  Army 
Signal  Center  and  School,  Fort  Monmouth,  New  Jersey,  during 
1968-1971)  showed  that  CAI  is  as  effective  or  better  than  con- 
ventional instruction  for  training  in  basic  electronics  (Longo, 
1972).  Those  initial  tests  were  conducted  with  the  IBM  1500 
Instructional  System,  using  the  IBM  Coursewriter  II  language. 
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Results  of  the  Fort  Monmouth  tests  are  described  elsewhere  In 
this  report;  results  of  the  Fort  Gordon  tests  were  published 
after  this  report  was  completed  (see  Seidel,  Rosenblatt,  Wagner, 
Schulz  and  Hunter,  1978).  , 

B . 2 PLATO  IV 

Development  of  the  PLATO  system  (Programmed  Logic  for  Auto- 
matic Teaching  Operations)  began  in  i960  under  the  leadership  of 
Donald  Bitzer  at  the  Computer-based  Education  Research  Labora- 
tory, University  of  Illinois  (called  Coordinated  Science  Labora- 
tory until  1967).  (See  Computer-based  Education  Research  Lab- 
oratory, 1977;  Smith  and  Sherwood,  1976;  Lyman,  1977).  PLATO 
IV,  the  current  version,  uses  a large  central  computer  (CDC 
CYBER  74)  at  CERL  which  supports  950  terminals  at  about  150 
locations  throughout  the  United  States  and  one  in  Sweden.  Other 
PLATO  systems  are  located  at  Control  Data  Corporation,  Arden 
Hills,  Minnesota,  and  at  Florida  State  University,  Tallahassee, 
Florida.  The  basic  architecture  of  the  PLATO  IV  system  can 
support  up  to  1008  terminals;  a Computer  Interface  Unit  controls 
data  communication  between  the  central  computer  and  up  to  32 
site  controllers,  each  of  which  can  support  up  to  32  terminals 
via  direct  connection  or  telephone  line.  The  PLATO  terminal 
contains  a touch-sensitive  display  panel,  keyboard,  and  micro- 
fiche projector;  It  can  also  control  various  multi-media  devices 
that  are  attached  to  It.  PLATO  was  developed  with  the  support 
of  the  National  Science  Foundation,  the  Office  of  Naval  Research, 
the  Defense  Advanced  Research  Projects  Agency,  the  National 
Institute  of  Education,  and  Control  Data  Corporation.  Control 
Data  Corporation  now  offers  CDC  PLATO  and  PLATO  Author  Language 
on  a commercial  basis;  these  are  production  versions  of  the 
PLATO  system  and  TUTOR  language  developed  at  CERL. 
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The  PLATO  IV  system  is  the  most  highly  sophisticated,  inter- 
active, individualized  instructional  system  currently  available 
(excluding  various  experimental  devices  still  under  development). 
It  provides  tutorial  inquiry,  drill  and  practice,  dialogue  modes 
of  instruction,  dynamic  simulation,  and  many  types  of  computa- 
tional services  and  games.  The  TUTOR  programming  language  con- 
tains over  250  commands  which  fall  into  five  large,  groups:  dis- 
play, calculation,  branching,  answer  judging,  and  data  collecting. 
A wide  variety  of  data  on  student  performance  with  various  seg- 
ments of  curricula  and  tests  are  available  to  Instructor  and 
management  personnel  for  analytical  and  management  purposes;  the 
system  supports  the  development  of  instructional  material.  The 
current  CDC  catalogue  lists  over  800  courses  and  games  that  are 
available  on  a commercial  basis. 

The  military  Services  have  evaluated  PLATO  IV  in  studies 
conducted  at  the  U.S.  Army  Ordnance  Center  and  School,  Aberdeen 
Proving  Ground;  Navy  Recruit  Training  Command,  Chanute  Air  Force 
Base;  and  other  locations.  These  are  summarized  elsewhere  in 
this  report.  The  National  Science  Foundation  supported  a demon- 
stration program  with  PLATO  IV  at  five  community  colleges  in  the 
area  of  Chicago;  a total  of  11  courses  in  selected  areas  of 
business,  biology,  chemistry,  English,  and  mathematics  were  de- 
veloped for  this  program  (Murphy  and  Appel,  1977).  In  1976, 
about  80  organizations  (12  military)  had  dedicated  communications 
lines  to  PLATO  (CERL);  PLATO  (CDC)  serves  many  universities  and 
commercial  organizations  as  well  as  its  own  learning  centers. 

(See  Computer-based  Education  Research  Laboratory,  1977,  p.  37.) 

In  addition  to  instruction,  the  PLATO  IV  system  presently 
provides  a broad  set  of  services,  such  as: 

• Electronic  mail. 

• On-line  communications,  including  text,  graphics, 
and  animation. 
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• Entertainment,  including  games,  musical  presentations, 
and  simulation. 

• Personal  services,  Including  medical,  financial, 
psychological,  educational  and  career  planning. 

• Research  computation. 

• Gn-line  research.  Physical  experiments  are  con- 
trolled by  the  PLATO  terminal,  and  analyzed  results 
are  displayed  graphically  in  real  time.  In  addition, 
educational  and  social  research  can  be  conducted  on- 
line and  in  real  time. 

• Data  processing. 

• Information  retrieval. 

B . 3 TICCIT 

The  TICCIT  system  (Time-shared,  Interactive,  Computer- 
Controlled,  Information  Television)  was  developed,  starting  in 
1971,  by  the  MITRE  Corporation,  with  support  from  the  National 
Science  Foundation.  C.  Victor  Bunderson  at  the  Institute  for 
Computer  Uses  in  Education,  Brigham  Young  University  (previously 
at  the  University  of  Texas),  and  M.  David  Merrill,  at  Western 
Montana  College  (previously  at  Brigham  Young  University)  were 
closely  associated  with  this  development,  primarily  using  courses 
in  freshman-level  mathematics  and  English.  Hazeltine  Corporation 
has  offered  TICCIT  on  a commercial  basis  since  1976. 

TICCIT  was  designed  to  provide  complete  courses  of  individ- 
ualized instruction  via  computer  on  a lower  cost  basis  than 
appeared  possible  with  existing  PLATO  and  IBM  systems.  Wherever 
possible,  TICCIT  used  commercially  available  rather  than  specially 
designed  components.  The  basic  system  uses  two  mini-computers  to 
support  up  to  128  terminals  and  maintain  records  for  up  to  3000 
students.  One  computer  serves  as  a main  processor,  the  other  as 
a terminal  processor  (both  are  Data  General  Nova  800).  The  ter- 
minals consist  of  a color  TV  receiver,  teletypewriter  keyboard, 
function  keys,  and  a light  pen;  graphic  and  audio-visual  material 
can  also  be  presented.  As  in  all  CA1  and  CMI  systems,  the  student 
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controls  the  pace  of  instruction.  TICCIT  courseware  is  struc- 
tured on  a modular  basis;  within  each  segment  of  instruction, 
the  student  can  select  material  to  be  presented  at  different 
levels  of  complexity  and  in  formats  that  provide  either  the 
basic  rule,  examples,  or  practice.  The  TICCIT  system  provides 
authors  with  one  instructional  strategy  (that  of  learner  control) 
in  order  to  simplify  the  task  of  programming;  this  differs  from 
PLATO  where  TUTOR  offers  several  types  of  instructional  pro- 
cedures, e.g.,  inquiry,  dialogue,  and  simulation. 

The  National  Science  Foundation  supported  a demonstration 
program  with  TICCIT  at  two  community  colleges;  Northern  Virginia 
Community  College  (Alexandria  Campus),  and  Phoenix  College  of 
the  Maricopa  County  Community  College  District,  Arizona;  the 
courses  selected  for  evaluation  were  mathematics  and  English 
(Alderman,  1978).  Other  TICCIT  installations  are  at  the  Model 
Secondary  School  for  the  Deaf  at  Gallaudet  College  in  Washington, 
D.C.,  and  at  Brigham  Young  University.  In  the  Department  of 
Defense,  TICCIT  has  been  used  on  an  experimental  basis  to  train 
tactical  coordinators  for  anti-submarine  warfare  in  the  S-3A 
aircraft  at  the  Naval  Air  Station,  North  Island,  San  Diego;  a 
mobile  system  was  installed  for  evaluative  purposes  at  the  Air 
University,  Maxwell  Air  Force  Base,  Montgomery,  Alabama. 

B . 4 AIR  FORCE  ADVANCED  INSTRUCTIONAL  SYSTEM 

Planning  for  the  Advanced  Instructional  System  (AIS)  at 
Lowry  Technical  Training  Center,  Lowry  Air  Force  Base,  Denver, 
Colorado,  started  In  February  1969,  when  the  Air  Force  Human 
Resources  Laboratory  (Technical  Training  Division)  published  a 
plan  for  the  development  of  a computer-managed,  computer-assisted 
Instructional  system.  The  system  was  developed  as  a computer- 
managed  instructional  system  and  tested  under  a contract  with 
the  McDonnell  Douglas  Astronautics  Company,  May  1973  to  De- 
cember 1977.  The  AIS  was  designed  to  be  a prototype  system; 
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it  incorporates  capabilities  for  research,  development,  test, 
and  evaluation  that  might  not  be  needed  in  an  operational  system 
(McDonnell  Douglas  Corporation,  1977a,  1977b). 

Current  (April  1978)  capabilities  of  the  AIS  are  as  fol- 
lows : 

Training  load 

No.  of  courses 
No.  of  students  per  day 

Hardware 

Computer 

Interactive  terminals 
Management  terminals 
Student  carrels 
Media  devices 

Media  allocation 

Printed  materials 

Audio/Visual  Presentation/ 

Illustrated  Text 

CAI  (used  for  management,  re 
search  and  course  develop- 
ment) 

AIS  provides  the  following  functions  common  to  most  CMI 
systems : 

• Printed  feedback  to  students  of  total  score  on  tests 
and  of  objectives  failed  on  tests 

• Printed  assignment  to  next  lesson.  Including  resources 
required 

Learning  center  rosters  and  individual  student  progress 
reports 


3000 

("500  with  additional 
terminals ) 

CDC  CYBER  73-16 
50 
11 
847 
500 

60 % 

38% 

2% 
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• Resource  management,  including  material  in  learning 
centers,  use  of  carrels,  audio-visual  devices  and  remote 
terminals 

• Advising  instructors  about  students  whose  results  on 
preassessment  tests  indicate  potential  problems 

• Displaying  or  printing  student  course  and  preassess- 
ment records  for  counseling 

• Providing  course  evaluation  and  test  item  evaluation 
summaries 

• Student  Progress  Management  (SPM):  SPM  predicts  a tar- 
get completion  time  for  each  student  for  each  block 
and  for  the  entire  course.  The  predictions  are  based 
on  the  student's  aptitude,  ability,  and  performance; 
students  and  instructors  receive  a daily  feedback  on 
each  student's  progress  toward  the  target  completion 
times.  The  purpose  of  SPM  is  to  pace  each  student  to 
work  at  a rate  judged  to  be  within  his  capability . 

• Individualized  Instruction  Assignment  ( 1 1 A ) : IIA  as- 
signs individual  students  to  alternative  modules  of  in- 
struction for  a lesson  in  order  to  achieve  maximum  pro- 
gress by  each  student.  An  adaptive  decision  process 
considers  the  Individual  characteristics  and  past  per- 
formance of  each  student  (preassessment  and  within- 
course  data),  his  current  placement  In  the  course  hier- 
archy and  the  availability  of  Instructional  resources. 
Each  student  is  assigned  to  those  modules,  among  the 
available  alternatives,  which  the  algorithm  predicts 

he  will  complete  in  the  shortest  time.  Three  methods 
of  making  this  assignment  were  tested:  a regression 
model,  "learner's  choice"  and  an  heuristic  method, 

(i.e.,  assignment  based  on  logical  rules).  IIA  is  a 
capability  unique  to  AIS  at  present. 
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Most  of  the  effort  required  to  develop  AIS  was  needed  to 
convert  four  courses  from  conventional  instruction  to  self- 
paced  form  suitable  for  support  by  CMI , These  four  courses 
were  selected  to  represent  a cross  section  of  all  technical 
training  and  a wide  range  of  student  aptitudes  and  abilities 
in  the  Air  Force.  These  courses  account  for  about  25  percent 
of  the  total  training  load  at  Lowry  Technical  Training  Center. 

Average  course  length  and  number  of  graduates  in  these 
courses  for  FY  1978  were: 


Courses  on  AIS 


Average  Number  of  Graduates 
Length  FY  ].978 

(weeks)  (Projected)  (Actual ) 


Inventory  Management  (IM) 

Materiel  Facilities  (MF) 

Precision  measuring  equipment  (PME) 
Weapons  Mechanic  (WM) 

Totals 


7 

3000 

2492 

6 

900 

743 

32 

600 

659 

13 

3009 

1514 

7500 

5408 

B . 5 NAVY  COMPUTER  MANAGEO  INSTRUCTION 

The  development  of  the  Navy's  Computer  Managed  Instruction 
(CMI)  system  can  be  traced  directly  to  work  started  in  1967  by 
G.  Douglas  Mayo,  then  on  the  staff  of  the  Chief  of  Naval  Air 
Technical  Training,  Millington,  Tennessee  (Kerr,  1978;  Middleton, 
Papettl  and  Mlchell,  1974).  At  that  time,  it  appeared  that  com- 
puter-assisted instruction,  such  as  provided  by  PLATO  or  the 
IBM  1500  Instructional  System,  would  be  effective  in  the  sense 
of  saving  student  time.  However,  it  appeared  that  implementa- 
tion of  CAI  systems  in  the  Navy  would  be  too  costly.  Mayo's 
premise  was  that  instruction  in  the  Navy's  technical  training 
courses  should  be  revised  from  conventional  to  individualized 
formats  and  that  computers  should  be  used  to  manage  but  not  to 
deliver  instruction. 
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The  Navy  began  to  implement  its  CMI  system  in  1973  and  ex- 
pects to  complete  its  installation  by  1980.  The  system  will 
handle  16,000  students  in  2b  separate  schools  at  six  Navy  train- 
ing centers;  there  were  6,000  students  in  11  schools  at  five 
training  centers  in  1978.*  Each  "learning  center"  (an  area  for 
about  100  students  in  a training  center)  has  an  optical  test 
scanner  (OPSCAN  17)  and  a General  Electric  Terminet  1200  key- 
board/printer. Each  school  has  access  to  a remote  batch  terminal, 
with  high-speed  printer  and  card  reader,  which  serves  various 
management  functions,  such  as  daily  progress  reports,  class 
rosters,  and  the  like.  The  schools  are  linked  to  a central  pro- 
cessor (Honeywell  Series  60,  level  66  computer)  located  at  the 
Management  Information  and  Instructional  Systems  Activity, 
Millington,  Tenn. 

Based  on  actual  student  loads,  cost-avoidance  savings  were 
estimated  to  be  $9  million  to  $10  million  per  year  from  FY  1975 
to  FY  1977;  they  are  expected  to  continue  at  the  latter  rate 
when  the  system  is  fully  implemented.  The  initial  savings  result 
largely  from  reductions  in  student  loads  because  of  improved 
management  of  student  time  by  CMI;  reductions  in  support  billets 
are  expected  to  occur  in  the  future.  Acquisition  of  the  system 
will  cost  $23.5  million  in  automatic  data  processing  equipment 
alone;  the  development  of  courseware  represents  ar.  additional 
cost.  Expansion  of  the  system  is  contingent  on  the  rate  at 
which  courses  can  be  individualized.  Instructional  Program 
Development  Centers  have  been  established  to  develop  and  maintain 
these  courses. 

In  addition  to  the  CMI  system  centered  at  Millington,  the 

' ’ 

Navy  has  CMI  systems  at  the  Naval  Air  Station,  Lemoore,  Cali- 
fornia (VA-122)  and  the  Naval  Air  Station,  Miramar,  California 
(VF-12^);  these  are  part  of  the  Aviation  Training  £ ipport  System. 

■Based  on  data  provided  by  Chief  of  Naval  Technical  Training 
(Code  0153) , April  1978. 


B-12 


The  Automated  Shipboard  Information  Management  System  is  a test 
installation  of  a shipboard  computer  which  also  provides  CMI 
services  on  the  USS  GRIDLEY  (transferred  from  the  USS  DAHLGREN, 
NPRDC  1977).  The  Marine  Corps  had  an  installation  at  Twenty- 
Nine  Palms,  California  that  is  inactive  at  present  (1978)  pend- 
ing receipt  of  a\new  computer.  Use  of  a communications  satellite 
has  been  considered  to  link  the  central  CMI  computer  at  Millington 
with  ships  at  sea  to  provide  CMI  for  training  personnel  away  from 
schools  (Polcyn,  1977). 

The  Navy  has  also  supported  development  of  Computer-Aided 
Instruction  Study  Management  System  (CAISMS)  which  uses  PLATO 
IV  to  give  reading  assignments,  to  give  tests  on-line,  and  to 
maintain  student  records.  These  functions  could  also,  of  course, 
be  performed  less  expensively  by  using  mini-computers  rather  than 
PLATO.  (Alessi,  Anderson,  Anderson  et  al.t  197^ , Nievergelt, 
Alessi,  and  Montague,  1978). 

More  recently,  CAISMS  was  evaluated  in  a Navy  technical 
training  course.  A conventionally  taught  section  in  an  interior 
communications  course  was  augmented  with  CAISMS;  this  was  in- 
tended to  provide  adjunct  Instructional  activities,  so  that 
students  could  more  appropriately  manage  their  study.  There  is 
potential  for  considerable  cost  savings  if  courses  are  configured 
to  take  advantage  of  the  flexibility  offered  by  computer  manage- 
ment (NPRDC  Technical  Report  "A  Computer-Based  Study  Management 
System:  Implementation  and  Evaluation  in  a Navy  Technical  Train- 

ing School,"  in  preparation). 

Also,  CAISMS  has  been  reprogrammed  to  run  on  a mini-computer, 
and  in  that  configuration  has  managed  the  study  of  over  4000 
students  in  a v/eek  of  a Navy  technical  course.  Reports  describ- 
ing this  implementation,  and  its  cost,  are  being  prepared  at  the 
time  of  this  writing  (December  1978). 
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B.6  STAND-ALONE  SYSTEMS 


Every  computer-based  instructional  system  needs  terminals 
to  deliver  instruction  (as  in  CAI)  or  to  score  tests  (as  in  CMI). 
From  the  student’s  point  of  view,  it  hardly  matters  whether  the 
terminal  is  supported  by  a computer  that  is  located  in  the  termi- 
nal or  elsewhere.  By  definition,  a "stand-alone"  instructional 
system  contains  a terminal  and  computer  in  one  unit  which  needs 
only  external  power  to  operate.  Two  stand-alone  CAI  systems 
that  have  been  evaluated  by  the  military  Services  are  described 
here.  They  now  are  estimated  to  cost  between  $30,000  and  $50,000 
per  unit,  without  courseware.  Interest  in  stand-alone  CAI  sys- 
tems will  probably  increase  when,  as  is  often  predicted,  the 
home  entertainment  market  will  make  micro-computers,  video 
storage  discs,  and  solid-state  memories  available  at  lower  cost. 

Stand-alone  systems  seem  promising  for  training  in  loca- 
tions away  from  schools  where  there  are  relatively  small  student 
loads,  few  or  no  instructors  available  on  site,  and  a demand  ex- 
ists for  a large  variety  of  courses.  Stand-alone  systems  could 
provide  and  guide  instruction  and  also  provide  administrative 
information,  such  as  student  progress  and  courses  completed,  in 
computer-compatible  form,  for  communication  to  central  personnel 
data  files. 

6.6.1  Lincoln  Terminal  System 

The  Lincoln  Terminal  System  is  a self-contained,  interac- 
tive, computer-based  training  system  developed  by  Lincoln  Labo- 
ratory, MIT,  Lexington,  Massachusetts.  The  latest  version,  LTS-5 
(Butman,  1977)  uses  microfiche  to  store  and  project  vieual  images 
in  a conventional  manner;  the  flche  can  also  store  audio  messages 
to  accompany  each  visual  frame  (up  to  750  microfiche,  each  with 
12  audlo/visual  pairs  of  frames  per  fiche).  The  user  interacts 
numbers  and  function  keys.  A teletypewriter  and  touch  panel 
could  be  added  to  the  system  but  they  are  not  in  the  present 
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version.  The  LTS  processor  supports  several  standard  author 
programs,  which  may  be  either  branching  or  nonbranching  In  form. 
The  system  is  designed  particularly  to  teach  facts,  principles, 
and  computational  skills. 

Only  a limited  number  of  LTS  terminals  have  been  built 
(about  UO) . The  system  has  been  tested  at  Keesler  Air  Force 
Base,  Biloxi,  Mississippi  (air  traffic  control  operator  course 
and  an  electronics  principles  course)  and  at  Lincoln  Laboratory 
(digital  systems  engineering;  see  Butman  and  Frick,  1972; 

Butman,  1975;  Butman  and  Kunze,  1976). 

B.6.2  General  Electric  Training  System 

The  General  Electric  Training  System  (GETS)  is  another  self-  ^ 
contained,  automated,  interactive,  instructional  system  (Rupp, 

1976;  General  Electric  Ordnance  Systems,  1976).  The  terminal 
contains  a plasma  display  panel,  teletypewriter  keyboard,  func- 
tion keys,  sonic  pen,  and  a random-access,  35-mm  slide  projector 
(80  slldes/tray ) . Floppy  discs  are  used  for  lesson  preparation 
and  playback.  The  plasma  screen  and  slides  can  be  used  for  in- 
teractive training,  e.g.,  using  text,  simulated  control  panels, 
or  circuit  diagrams.  To  date,  GETS  has  been  used  for  training 
on  operating  and  maintenance  procedures  for  the  TRIDENT  weapon 
control  system  at  the  Guided  Missile  School,  Dam  Neck,  Virginia 
and  the  Fleet  Ballistic  Missile  Training  Center,  Charleston, 

South  Carolina;  it  is  scheduled  for  use  at  the  Fleet  Ballistic 
Missile  Training  Center,  Bangor,  Washington.  There  will  be 
about  25  GETS  units  in  the  TRIDENT  program.  The  TRIDENT  program 
relies  heavily  on  the  use  of  tactical  equipment  and  equipment 
simulation  for  training  purposes.  GETS  will  be  used  primarily 
to  handle  peak  training  loads  that  exceed  the  capacity  of  the 
available  training  equipment  (called  "laboratory"  training  in 
this  program).  GETS  is  viewed  as  cost-effective  for  procedural 
training  in  the  TRIDENT  program  compared  to  the  cost  of  acquir- 
ing additional  tactical  equipment.  Current  training  loads  on 
GETS  are  small,  but  larger  loads  are  expected  to  occur  in  April 
1981. 
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APPENOIX  C 


EFFECTIVENESS  OF  COMPUTER-ASS  I STED  INSTRUCTION 
IN  NONMILITARY  TRAINING 

C.l.  INTRODUCTION 

This  appendix  considers  briefly  the  effectiveness  of  CAI 
and  CMI  instruction  in  nonmilitary  settings,  primarily  in 
schools  and  colleges.  Education  in  schools  and  colleges  dif- 
fers from  military  training  in  one  major  characteristic  of  in- 
terest here:  students  in  military  training  receive  pay  and  al- 
lowances while  they  are  being  trained,  while  those  in  schools 
and  colleges  do  not.  This  means  that  reducing  the  time  spent 
at  school  could  reduce  the  cost  of  military  training  while  no 
such  incentive  exists  in  schools  and  colleges  (at  least  at 
present).  Other  distinctions  between  schools  and  colleges  and 
military  training  might  also  be  drawn  with  respect  to  such 
factors  as  subject  matter,  tenure  of  instructors,  and  the  rela- 
tionship of  training  to  Jobs  and  careers;  however,  these  are 
not  critical  to  the  present  discussion. 

The  effectiveness  of  CAI  and  CMI  in  schools  and  colleges 
has  been  the  subject  of  many  excellent  books  and  reviews  such 
as  the  following: 

Kearsley  (1975);  Seidel  and  Rubin  (1977);  Salomon 
and  Clark  (1977);  Davisson  and  Bonello  (1976);  Froomkin, 
Jamison,  and  Radver  (1976);  Levien  (1972);  Goldstein 
(197*0;  Edwards,  Norton,  et  at.  (197**);  Jamison,  Suppes 
and  Wells  (197**);  and  Baker  (1978). 

Most  of  these  are  concerned  with  CAI  which  has  received 
more  attention  in  schools  and  colleges  than  has  CMI.  The  issue 
of  effectiveness  here  is  almost  entirely  on  student  achievement, 
that  is,  tne  amount  of  course  materials  acquired  as  measured  by 
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tests;  some  attention  has  been  given  to  the  effect  of  CAI  on 
academic  attrition. 

The  literature  cited  above  suggests  that  CAI  is  an  effec- 
tive supplement  to  regular  instruction  at  the  elementary  school 
level;  used  as  a replacement,  it  is  about  as  effective  as  con- 
ventional instruction  at  the  secondary  school  and  college  levels. 
The  drill  and  practice  and  simulation  modes  of  CAI  are  at  least 
as  effective  as  conventional  instruction;  some  studies  suggest 
that  they  are  more  effective  than  conventional  instruction; 
the  results  are  equivocal  for  the  tutorial  and  problem-solving 
modes.  The  apparent  differences  in  the  effectiveness  of  various 
CAI  modes  may  be  the  result  of  improper  comparisons  because, 
for  example,  CAI  drill  and  practice  is  generally  used  to 
supplement , while  the  other  modes  are  used  to  replace  conven- 
tional Instruction.  Limited  data  suggest  that  CAI  can  reduce 
the  time  required  for  learning.  According  to  Baker  (1978), 
about  30  CAI  systems  are  being  used  in  academic  environments  at 
all  levels  of  education. 

The  National  Science  Foundation  supported  two  large-scale 
studies  that  evaluated  the  effectiveness  of  the  PLATO  IV  and 
TICCIT  systems  In  teaching  basic  courses  at  community  colleges. 
Both  of  these  CAI  systems  have  been  used  experimentally  by  the 
military  Services.  These  studies  are  summarized  briefly  here 
because  of  their  importance  and  their  potential  relevance  to 
the  effectiveness  of  PLATO  IV  and  TICCIT  in  military  training. 

The  studies  do  not  consider  the  amount  of  time,  if  any,  saved 
by  students,  the  effectiveness  of  the  particular  coursewares 
in  distinction  to  the  delivery  systems,  or  the  cost-effective- 
ness of  these  two  systems.  A survey  of  student  activities  in 
the  TICCIT  „ij  p /ides  an  indirect  assessment  of  how  much 
time  students  sper  in  conventional  and  CAI  instruction. 
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C.2.  EFFECTIVENESS  OF  PLATO  IV  AT  COMMUNITY  COLLEGES 


The  effectiveness  of  the  PLATO  IV  CAI  system  was  evaluated 
at  five  community  colleges  in  the  Chicago  area,  1972  to  1976. 
(Murphy  and  Appel  1977;  Computer-based  Education  Research  Labora- 
tory, 1977).  Most  of  the  time  during  this  period  was  required 
to  develop  course  materials  and  tests  and  to  conduct  prelimin- 
ary evaluations.  The  final  evaluation  was  conducted  during 
two  semesters  (1975-1976)  in  162  classes  in  five  subject-matter 
areas  (accounting,  biology,  chemistry,  English,  and  mathematics) 
at  four  of  the  colleges.  Most  comparable  PLATO  and  non-PLATO 
courses  were  taught  by  the  same  instructors,  thereby  holding 
constant  the  possible  influence  of  instructors  on  student 
achievement,  attrition,  and  attitudes.  Instructors  who  parti- 
cipated in  the  test  were  not  required  to  use  PLATO  IV  for  any 
specified  amount  of  time  or  for  any  specified  material;  rather, 
they  used  PLATO  IV  in  various  ways  to  replace,  supplement,  or 
reinforce  classroom  instruction.  Average  student  use  of 
PLATO  IV  varied  from  a few  minutes  to  more  than  20  hours  for 
individual  students;  in  terms  of  courses,  use  of  PLATO  IV 
varied  from  1 to  12  hours  per  course  for  126  courses.  The  per- 
missive approach  made  PLATO  IV  very  acceptable  to  the  faculty 
but  it  complicates  and  makes  it  more  difficult  to  extrapolate 
the  findings  of  this  evaluation  to  the  more  highly  controlled 
environments  of  most  military  studies  where  students  and  in- 
structors had  no  option  to  shift  back  and  forth  between  CAI 
and  conventional  instruction. 

Bearing  in  mind  the  way  in  which  the  evaluation  was  per- 
formed, the  following  findings  are  significant: 

• Student  achievement  on  PLATO  IV  was  about  the  same  as 
that  for  regular  classroom  instruction 

• PLATO  IV  produced  no  noticeable  effect  on  student  at- 
trition 

• Student  and  faculty  attitudes  to  the  use  of  PLATO  IV 
were  generally  favorable 
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• The  development  of  curriculum  materials  was  a diffi- 
cult undertaking.  The  test  was  postponed  for  one  year 
while  additional  staff  were  added  to  the  project  to 
develop  more  PLATO  IV  lessons. 

C.3.  EFFECTIVENESS  OF  TICCIT  AT  COMMUNITY  COLLEGES 

The  effectiveness  of  TICCIT  was  evaluated  by  comparing  the 
performance  of  students  in  six  mathematics  and  three  English 
courses  instructed  by  TICCIT  or  conventional  lecture,  textbook 
and  discussion  (Alderman,  1978).  The  study  was  conducted  in 
1975-1976  at  Phoenix  College,  Arizona,  and  the  Alexandria  cam- 
pus of  the  Northern  Virginia  Community  College.  Two  years  were 
required  to  develop  the  course  materials  and  achievement  tests 
and  to  train  the  staff  involved  in  the  evaluation.  Over  2600 
students  were  enrolled  in  the  TICCIT  courses  and  3000  in  the 
lecture  courses;  ir.  addition,  about  300  students  at  Alexandria 
took  programmed  courses  in  mathematics  without  computer  support. 

A measure  of  effectiveness  which  turned  out  to  be  important  was 
the  percentage  of  students  who  completed  the  course  under  each 
mode  of  instruction. 

Instruction  by  TICCIT  was  at  least  as  effective  as  by 
lecture  or  by  programmed  material.  Students  instructed  by 
TICCIT  had  higher  test  scores  (by  about  10  percent)  than  those 
instructed  by  lecture  in  nine  of  twelve  mathematics  courses  and 
in  four  of  seven  English  courses;  where  TICCIT  test  scores  were 
lower,  the  differences  were  quite  small. 

Some  of  the  differences  that  favor  TICCIT  may  be  attribut- 
ed tc  the  related  finding  that  students  who  completed  the  TICCIT 
classes  were  more  highly  qualified  than  those  in  the  conventional 
classes.  The  much  lower  course  completion  rates  on  TICCIT, 
compared  to  the  lecture  course,  are  certainly  related  to  the 
finding  that  only  the  mere  qualified  students  completed  the 
TICCIT  course. 
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A surprise  finding  is  the  low  completion  rates  for  students 
taking  courses  or,  TICCIT,  particularly  mathematics.  The  over- 
all completion  rates  for  mathematics  were  16  percent  for  TICCIT, 
50  percent  for  lecture,  and  20  percent  for  programmed  instruc- 
tion. When  the  data  are  adjusted  to  include  students  who  did 
not  complete  the  course  or  who  withdrew  and  reenrolled,  i.e 
completed  the  course  in  other  semesters),  the  adjusted  comple- 
tion rates  for  mathematics  on  TICCIT  are  comparable  to  those 
for  other  methods  of  instruction.  Completion  rates  were  higher 
for  English  than  for  mathematics,  but  instruction  by  TICCIT  re- 
sulted again  in  lower  completion  rates  than  by  lecture. 

This  study  shows  that  instruction  in  college  algebra  and 
English  composition  by  TICCIT  produces  end-of-course  results 
i.e.,  test  scores)  that  are  equal  to  or  higher  than  those  as- 
sociated with  conventional  Instruction.  TICCIT  appears  more 
effective  for  algebra  than  English.  These  findings  may  be  an 
artifact  due  to  the  dropout  of  the  poorer  students  before  the 
end  of  the  course.  TICCIT  instruction  appears  more  favorable 
for  higher-aptitude  than  for  average  or  lower-aptitude  students ; 
very  few  of  the  latter  completed  the  courses  on  TICCIT  in  this 
study.  The  TICCIT  study  is  one  of  the  few  that  have  examined 
CAI  instruction  in  entire  courses,  under  stabilized  conditions, 
and  on  a large  scale;  its  use  in  more  than  one  location  is  also 
unusual.  However,  there  is  little  reason  to  believe  that  the 
permissive  atmosphere  of  a community  college  with  respect  to 
failure  to  complete  courses  provides  a basis  for  comparison  with 
military  training. 

The  amount  of  time  required  to  complete  courses  by  TICCIT 
cr  by  conventional  instruction,  an  important  issue  in  military 
and  Industrial  training,  was  not  addressed  directly  in  this 
study.  However,  the  report  contains  survey  data  on  how  many 
hours  students  said  they  spent  on  course  activities  out  of 
class.  An  analysis  of  these  data  suggests  that  students  on 
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TICCIT  completed  these  courses  in  less  total  time  than  did 
those  in  conventional  courses.  This  analysis  appears  in  the 
following  section. 

C.4.  ANALYSIS  OF  TIME  SPENT  BY  STUDENTS  ON  TICCIT  AND  IN 
CONVENTIONAL  INSTRUCTION  AT  COMMUNITY  COLLEGES 

Two  types  of  data  reported  by  Alderman  (1978)  can  be  com- 
bined to  estimate  the  total  amounts  of  time  spent  by  students 
to  complete  five  courses  instructed  by  TICCIT  or  conventional 
lecture  and  discussion.  The  data  consist  of  (1)  student  con- 
tact hours  (time  spent  on  TICCIT  or  in  c.  ss)  and  (2)  a Student 
Activity  Survey  (in  which  students  report  how  much  tine  they 
spent  on  course-related  activities  in  addition  to  the  time  in 
class).  A description  of  these  data  and  how  they  were  used  to 
estimate  total  time  spent  by  students  on  TICCIT  and  In  conven- 
tional instruction  follows.  The  results  are  shown  in  Table  C-l. 

C . 4 . 1 Student  Contact  Hours 

Data  on  student  contact  hours  with  TICCIT  were  compiled 
from  records  kept  by  the  TICCIT  system;  data  on  hours  spent  in 
lectures  are  simply  the  result  of  scheduled  hours  for  each 
class.  Thus,  for  eight  courses,  a direct  comparison  can  be 
made  of  the  amount  of  time  spent  by  students  who  used  TICCIT 
and  by  students  in  class  with  conventional  instruction.  In 
five  of  the  eight  comparisons,  students  on  TICCIT  spent  less 
time  during  the  course  than  did  those  in  class.  These  data 
are  probably  highly  reliable  but  they  do  not  include  time  spent 
on  course  work  out  of  class. 

C . 4 . 2 Student  Activity  Survey 

Data  on  student  activities  were  collected  only  in  five 
courses.  A survey  of  student  activities  included  the  following 
questions  about  time  spent  by  students  in  each  course: 
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TABLE  C-l.  AMOUNT  OF  TIME  NEEDED  TO  COMPLETE  COURSES  ON  TICCIT 
ANO  CONVENTIONAL  INSTRUCTION,  1976 
Source:  Alderman  (1978) 
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"Approximately  how  many  hours  per  week  did  you  spend: 

(1)  working  on  the  TICCIT  system/attending  classes 
for  this  course? 

(2)  in  small  group  discussions  about  this  course 
(outside  of  class)? 

(3)  doing  work  for  this  course  on  your  own  away  from 
TICCIT/working  on  homework  assignments? 

(4)  in  total  for  this  course?" 

The  students'  responses,  In  hours  per  week,  were  multi- 
plied by  the  number  of  weeks  In  the  semester  (15  weeks)  or 
quarter  (10  weeks)  to  get  the  total  times  shown  in  the  table. 

In  four  of  the  five  con.par  Isons , students  on  TICCIT  say  they 
spent  less  total  time  to  complete  the  course  than  did  those  in 
conventional  lecture  and  discussion.  Naturally,  these  compari- 
sons are  based  on  the  students'  impressions  and  attitudes  and 
cannot  be  verified. 

C.4.3  Calculated  Total  Times 

An  estimate  of  the  total  time  spent  by  students  in  each 
course  was  made  by  combining  the  documented  tire  spent  on 
TICCIT  or  in  lecture  (Columns  2 and  3)  with  students'  reports 
of  time  spent  out  of  class  on  course  work  (part  of  the  total 
data  shown  in  Columns  5 and  7,  based  on  detailed  data  in  the 
Appendix  to  Alderman's  report).  In  four  of  the  five  compari- 
sons, students  on  TICCIT  appear  to  have  spent  less  total  time 
j r.  the  course  than  did  those  in  the  conventional  classes; 
the  reverse  effect  occurs  in  one  comparison.  In  three  of  the 
five  cases,  students  on  TICCIT  appear  to  have  spent  about  half 
the  total  time  needed  to  complete  the  course  that  those  in 
conventional  classes  did. 

Although  there  is  no  way  to  assess  the  reliability  of  these 
data  on  the  times  required  to  complete  courses  on  TICCIT  or  In 
conventional  classes,  they  are  the  only  data  we  have.  The  ap- 
parent time  savings  on  TICCIT  would  apply  only  to  students  who 
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completed  the  course  (16  percent  on  TICCIT,  50  percent  on  lec- 
ture, as  reported  by  Alderman).  Those  who  completed  courses 
on  TICCIT  had  higher  pre-test  scores  than  their  lecture  counter- 
parts. Thus,  the  time  and  achievement  advantages  of  TICCIT  for 
the  students  described  here  do  not  apply  to  the  majority  of 
students  who  were  unable  to  complete  a course  on  the  TICCIT 
program. 
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APPENDIX  D 


SUMMARY  OF  STUDIES  EVALUATING  COMPUTER-ASSISTED 
AND  COMPUTER-MANAGED  INSTRUCTION  IN  MILITARY  TRAINING 


LIST  OF  TABLES 


1.  Studies  of  computer-assisted  instruction  (IBM  1500)  at 
U.S.  Army  Signal  Center  and  School,  Pt . Monmouth,  New 
Jersey,  1968-1972. 

2.  Studies  of  computer-assisted  instruction  (IBM  1500)  at 
Navy  Basic  Electricity  and  Electronics  School,  San  Diego, 
California,  1970-1972. 

3.  Studies  of  computer-assisted  instruction  (PLATO  IV)  at  U.S. 
Army  Ordnance  Center  and  School,  Aberdeen  Proving  Ground, 
Maryland,  1975. 

4.  Studies  of  computer-assisted  instruction  (PLATO  IV)  at 
Navy  Basic  Electricity  and  Electronics  School,  Fleet 
Aviation  Specialized  Operational  Training  Group,  Pacific 
Fleet  and  Mess  Management  School,  San  Diego,  California, 
1975-1978. 

5.  Studies  of  computer-assisted  instruction  (PLATO  IV)  at 
Air  Force  School  of  Health  Care  Services,  Sheppard  AFB , 
Texas,  1977. 

6.  Studies  of  computer-assisted  instruction  (PLATO  IV)  at  Air 
Force  Chanute  Technical  Training  Center,  Chanute  AFB, 
Illinois,  1977. 

7.  Studies  of  computer-assisted  instruction  (Lincoln  Terminal 
System-3)  at  Air  Force  Kcesler  Technical  Training  Center, 
Keesler  AFB,  Mississippi,  1972-1973- 

8.  Studies  of  computer-assisted  instruction  (TICCIT)  at  Navy 
Squadron  VS-41,  North  Island  Naval  Air  Station,  San  Diego, 
California,  1978. 

9.  Studies  of  computer-assisted  instruction  (IDII0M  and  PLATO 
IV)  at  Navy  Guided  Missile  School,  Dam  Neck,  Virginia, 

1975- 

10.  Studies  of  computer-managed  Instruction  (Navy  Computer 

Managed  Instruction  System)  at  Naval  Air  Technical  Train- 
ing Center,  Millington,  Tennessee,  1975- 
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11.  Studies  of  computer-managed  Instruction  (Advanced  Instruc- 
tional System)  at  Air  Force  Lowry  Technical  Training 
Center,  Lowry  AFB,  Colorado,  1978. 


TABLE  0-1.  STUDIES  OF  COMPUTER-ASSISTED  INSTRUCTION  (IBM  1500)  AT  U.S 
ARMY  SIGNAL  CENTER  AND  SCHOOL,  FT.  MONMOUTH,  NEW  JERSEY,  1968-72 


TABLE  0-2.  STUDIES  OF  COMPUTER- ASSISTED  INSTRUCTION  (IBM  1500)  AT  NAVY  BASIC 


TABLE  D-4.  STUDIES  OF  COMPUTER-ASSISTED  INSTRUCTION  (PLATO  IV)  AT  NAVY  BASIC  ELECTRICITY 
AND  ELECTRONICS  SCHOOL,  FLEET  AVIATION  SPECIALIZED  OPERATIONAL  TRAINING  GROUP, 
PACIFIC  FLEET  AND  MESS  MANAGEMENT  SCHOOL,  SAN  DIEGO,  CALIFORNIA,  1975-1978 


Footnotes  et  end  of  title  (continued) 


TABLE  0-4.  (Continued) 


TABLE  D-5.  STUDIES  OF  COMPUTER- ASSISTED  INSTRUCTION  (PLATO  IV)  AT  AIR  FORCE 
SCHOOL  OF  HEALTH  CARE  SERVICES,  SHEPPARD  AFB.  TEXAS.  1977 
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TABLE  0-6.  STUDIES  OF  COMPUTER- ASSISTED  INSTRUCTION  (PLATO  IV)  AT  AIR  FORCE 
CHANUTE  TECHNICAL  TRAINING  CENTER,  CHANUTE  AFB,  ILLINOIS,  1977 


it Ion  saves  tine ; 

(amount  not  stated) 

PLATO  saved  10%  wore. 

30  terminals 

»“«'«  « 01  U61*.  (continue) 


TABLE  D-6.  (Continued) 
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TABLE  D-7.  STUDIES  OF  COMPUTER-ASSISTED  INSTRUCTION  (LST-36)  AT  AIR  FORCE  KEESLER 
TECHNICAL  TRAINING  CENTER,  KEESLER  AFB,  MISSISSIPPI,  1972-73 


TABLE  D-7.  (Continued) 


TABLE  0-9.  STUDIES  OF  COMPUTER-ASSISTED  INSTRUCTION  (1DII0M  AND  PLATO  IV)  AT 


APPENDIX  E 


SUMMARY  OF  DATA  ON  COSTS  OF  INSTRUCTION 

E.1  PROGRAM  DEVELOPMENT:  PROGRAM  DESIGN 

1.2  PROGRAM  DEVELOPMENT:  INSTRUCTIONAL  MATERIALS  FOR 

CONVENTIONAL  INSTRUCTION 

E . 3 PROGRAM  DEVELOPMENT:  INSTRUCTIONAL  MATERIALS  FOR 
INDIVIDUALIZED  INSTRUCTION 

E . 4 PROGRAM  DEVELOPMENT:  INSTRUCTIONAL  MATERIALS  FOR 
COMPUTER-BASED  INSTRUCTION 

E.5  PROGRAM  DELIVERY:  INSTRUCTION,  INCLUDING  INSTRUCTORS 

AND  INSTRUCTIONAL  SUPPORT  PERSONNEL  AND  STUDENT  PERSONNEL 

E . 6 PROGRAM  DELIVERY:  LABORATORY  EQUIPMENT 

E . 7 PROGRAM  DELIVERY:  MEDIA  DEVICES 

E . 8 INSTRUCTIONAL  PROGRAM  DELIVERY:  COMPUTER  SYSTEMS 


E.8.1 

PLATO  IV 

E.8.2 

TICCIT 

E.8.3 

Navy  CMI  System 

E.8.4 

IBM  1500 

E.8.5 

General  Electric  Training  System  (GETS) 

E.8.6 

Shipboard  Computer-Based  Instruction 

E.8.7 

System  Cost  Comparison 

E . 9 

PROGRAM 

DELIVERY:  COMMUNICATIONS 

E . 1 0 

PROGRAM 

DELIVERY:  MATERIALS 

E . 1 1 

PROGRAM 

DELIVERY:  FACILITIES 

• • 

• • » 

• • %••  • » • « 

E . 1 2 

PROGRAM 

MANAGEMENT  AND  ADMINISTRATION 

E.13 

STUDENT 

PERSONNEL 
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SUMMARY  OF  DATA  ON  COSTS  OF  INSTRUCTION 

This  appendix  presents  the  data  on  costs  of  instruction 
that  we  were  able  to  find  and  which  are  summarized  in  Chapter 
IV.  The  presentation  is  organized  according  to  type  or  func- 
tion of  resources  required  to  conduct  military  training.  These 
categories  are  identified  in  Table  E-l. 

The  data  are  shown  as  reported  in  the  literature.  Ho  ad- 
justments have  been  made  to  bring  cost  levels  to  a common  base 
period.  The  time  oeriods  in  which  costs  were  incurred  are 
generally  not  shown  in  the  source  documents  and  may  differ,  by 
varying  periods,  from  dates  of  publication.  The  use  of  standard 
indices,  such  as  wholesale  prices,  does  not  appear  appropriate 
to  adjust  all  costs  to  a common  base;  specialized  indices,  that 
are  not  available,  would  be  required  for  some  types  ol  resources, 
such  as  various  components  of  computer  systems. 

The  validity  of  individual  data  has  not  been  evaluated. 

Some  data  values  were  extracted  from  secondary  sources  that  did 
not  reference  original  sources.  Data  in  some  secondary  sources 
duplicated  information  already  available  in  primary  sources  and 
were  not  used;  however,  undetected  duplications  may  remain.  Data 
that  were  not  well  enough  described  to  be  interpreted  with  confi- 
dence have  been  excluded.  Wherever  a value  was  shown,  it  was 
assumed  to  be  based  on  historical  experience  unless  it  was 
specifically  described  as  a programmed  or  planned  value. 

No  references  to  costs  of  conventional  instruction  wex e 
Thl:  may  be  due  to  our  apprc  eh  to  the  literature. 

t-  5 


9 


found . 


TABLE  E-1.  RESOURCES  REQUIREO  FOR  VARIOUS  METHODS 
OF  INSTRUCTION  IN  MILITARY  TRAINING 


Resource  (Type  or  Function) 

Program  Development 
Program  Design 

Instructional  Materials:4  Conventional  Instruction 

Individualized  Instruction 
Programming 

first  Unit  Production13 
Computer-Based  Instruction 
Programming 
Ceding 

Program  Delivery 


Instruction:  Instructors 

Instructional  Support  Personnel0 
Equipment  and  Services:*1  Laboratory  (including  simulators) 

Media  Oevices 
Computer  Systems 
Commun  < ca  1 1 ons 

Materials  (including  consumables)6 
f ac  1 1 1 1 i esf 

Program  Management  and  Administration 

Student  Personnel:  Pay  Allowances 

Other  (PCS  . TOY , etc. ) 


NOTE:  * No  data  available. 

**  Included  in  discussion  of  Instruction:  Instructors  and  Instructional 
Support  Personnel  (Section  5). 

includes  revision. 
bMaster  copy. 

CA1 1 direct  personnel  not  included  in  other  categories. 

^Includes  all  hardware-related  costs : initial  (including  installation  and  checkout), 
modification,  and  replacement;  operation  and  maintenance,  lease  and  user  fees;  computer 
system  software;  etc. 

includes  copies  Of  instructional  materials  (books,  courseware  copies,  etc.). 
^Structures,  fixtures,  and  furnishings. 

’ ♦ * e*  •«-  « - » • . . . 
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Emphasis  was  placed  on  computer-based  instruction,  and  we  ex- 
pected that  it  would  be  evaluated  in  terms  of  its  alternatives 
(conventional  or  individualized  instruction).  This  was  not  the 
case.  Student  hours  required  by  computer-based  and  conventional 
instruction  were  frequently  compared;  however,  these  data  were 
not  converted  to  equivalent  dollar  costs.  Other  resources  needed 
in  computer-based  Instruction  were  simply  presented  in  dollar  or 
real  terms,  but  no  other  data  were  reported  on  the  costs  of  con- 
ventional instruction.  This  raises  an  obvious  question.  What 
good  does  it  do  to  know  the  cost  of  some  particular  version  of 
computer-based  instruction  if  little  is  Known  about  the  cost  of 
conventional  instruction  or  of  any  other  method  of  instruction 
to  which  it  might  be  compared? 

Several  studies  noted  that  significant  man-hours  are  asso- 
ciated with  Program  Design,  but  provided  no  further  information. 
Several  studies  noted  that  savings,  due  to  decreases  In  student 
hours,  represented  the  combined  impact  of  course  revision  and  a 
change  in  the  method  of  Instruction.  One  study  noted  an  expen- 
diture of  1*J  man-years  for  a course  revision  that  decreased  the 
length  by  50  percent  but  provided  no  information  regarding  what 
was  Involved  in  the  revision.  The  magnitudes  of  these  values 
amount  to  a strong  argument  for  considering  the  benefits  of 
course  revision  alone,  without  changes  in  instructional  method. 

The  cost-effectiveness  of  course  revisions  alone  should  be  eval- 
uated as  a competitor  of  CAI , CMI , and  individualized  Instruction. 

No  data  nor  any  discussion  was  found  regarding  Instruc- 
tional Program  Management  requirements,  and  only  one  was  cited 
that  discussed  other  Instructional  Support  Personnel  resources. 
This  can  be  understood  with  regard  to  CAI  where  all  applications, 
save  one,  have  been  experimental  programs  of  limited  duration. 

•However',  with)  'respect 'to  Othef  'Instructional  methods,  including 
CMI,  it  should  have  been  possible  to  develop  such  information. 

With  the  highly  organized  structure  of  military  training,  one 
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must  allow  that  both  Program  Management  and  Other  Instructional 
Support  may  account  for  a significant  share  of  total  program 
cost,  that  they  are  subject  to  analysis,  and  that  they  may  vary 
sufficiently  between  different  methods  to  have  a noticeable  im- 
pact on  cost-effectiveness. 

E.1  PROGRAM  DEVELOPMENT;  PROGRAM  DESIGN 

No  data  were  found  on  these  costs  of  instruction. 
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PROGRAM  DEVELOPMENT:  INSTRUCTIONAL  MATERIALS  FOR 
Co VENT  I ON AL  INSTRUCTION 


No  data  were  found  on  these  costs  of  Instruction. 
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PROGRAM  DEVELOPMENT:  INSTRUCTIONAL  MATERIALS  FOR 

Individualized  Instruction 


Development  of  instructional  materials  for  individualized 
Instruction  involves  two  distinct  functions  that  should  be  kept 
separate.  The  first  is  the  Instructional  Programming  (or  author- 
ship). The  second  is  the  First  Unit  Production  (or  master  copies) 
of  the  courseware  material.  Only  two  of  six  studies  reporting 
costs  maintained  this  distinction.  The  most  notable  feature  of 
these  data  is  their  wide  range,  Table  E-2.  For  master  copy  pro- 
duction the  variability  holds  both  between  different  types  of 
media  material  and  within  one  type.  Depending  upon  the  number 
of  silent  slides  or  printed  pages  that  might  comprise  an  hour  of 
Instruction,  the  data  indicate  a possible  range  between  a few 
hundred  dollars  to  over  $10,000  per  hour  of  Instruction.  The 
U.3.  Army  Ordnance  Center  and  School  (1975) > the  single  source 
of  information  on  authoring  requirements,  noted  values  of  tlO 
man-hours  per  Instructional  hour  for  sound-slides  and  280  man- 
hours per  instructional  hour  for  sound  motion  picture  or  TV. 

These  limited  data  and  the  limited  discussions  presented  in  the 
citations  make  it  impossible  to  understand  the  reaeons  for  t -he*  •»  • 
difference.  Table  E-3  displays  the  only  information  found  on 
courseware  material  reliability,  and  no  information  was  provided 
as  to  the  repal rability  or  repair  costs  of  failed  courseware. 
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TABLE  E-2.  INDIVIDUALIZED  INSTRUCTION:  INSTRUCTIONAL  MATERIALS 
DEVELOPMENT,  REQUIREMENTS  AND  COSTS 
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TABLE  E-3.  MEDIA  COURSEWARE  RELIABILITY 


— 

Material 

Uses 

Fa i 1 u res 

Super  8 Film 

1 .648 

7 

Video  Tape 

982 

18 

Filmstrip 

24,445 

221 

Audio  Tape 

25,154 

36 

Source:  AIS  Integrated  System  Test, 
October  1977. 


A variety  of  media  are  available  for  presentation  of  indi- 
vidualized instruction,  and  different  media  are  substitutes  for 
each  other  in  presenting  the  subject  matter  of  small  units  of 
instruction,  e.g.t  the  individual  lesson.  As  a result,  a course 
(or  segment  of  a course)  may  utilize  a mixture  of  media.  With 
the  variations  in  courseware  costs  noted,  different  mixes  of 
media  can  imply  sizeable  differences  in  course  costs.  However, 
determination  of  the  most  effective  media  mix  requires  a rather 
extensive  course  design  effort,  and  systematic  investigation  of 
alternative  course  designs  to  determine  cost  differences  asso- 
ciated with  these  mixes  is  an  expensive  process. 

E . 4 PROGRAM  DEVELOPMENT:  INSTRUCTIONAL  MATERIALS  FOR 

COMPUTER-BASED  INSTRUCTION 

Table  E-*4  displays  information  on  development  of  computer- 
based  instructional  materials.  Close  to  1,000  hours  of  material 
are  represented,  but  in  only  three  cases  were  more  than  ^0  hours 
produced  by  one  authoring  group.  The  striking  feature  of  the 
table  is  the  variability  of  the  data — approaching  an  order  of 
-.rvurftitiKjfih . ..  ...  

Several  studies  provided  data  on  man-hours  needed  sepa- 
rately for  authoring  and  coding.  For  CMI , authorship  v;as  the 
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dominant  requirement  and  accounted  for  90  percent  of  the  man- 
hours. For  CAI,  weighted  averages  indicate  that  the  require- 
ments for  each  function  are  roughly  equal,  with  52  percent  of 
the  total  attributed  to  authoring.  However,  wide  differences 
may  be  noted  with  authoring  accounting  for  as  little  as  11  per- 
cent and  as  much  as  75  percent. 

Instructional  materials  may  be  developed  in-house  by 
military  or  civilian  personnel  or  by  contract.  Military  per- 
sonnel would  come  from  the  more  senior  pay  grades,  e.y.,  E-5  or 
E-6)  with  pay  and  allowance  rates  near  $6  per  hour.  Costs  of 
contract  personnel  should  be  roughly  $30  per  hour  (Middleton, 
Papetti,  and  Micheli,  197^,  adjusted  to  1978  wage  levels).  On 
the  basis  of  these  hourly  labor  rates  and  the  man-hours  require- 
ments shown  in  Table  E-*4,  the  costs  of  courseware  development 
might  currently  be  estimated  as  high  as  $21,000  or  as  low  as 
$500  per  instructional  hour  for  CAI  and  as  high  as  $3,300  or  as 
low  as  $200  for  CMI . 

Grimes  (1975)  presented  the  only  analysis  of  programming 
requirements.  He  cites  8o  man-hours  per  Instructional  hour  as 
the  weighted  average  of  16  programmers,  all  of  whom  were  either 
students  or  project  personnel  at  the  University  of  Illinois.  On 
an  Individual  programmer  basis,  man-hour  expenditures  per  in- 
structional hour  averaged  182.  The  difference  in  averages  in- 
dicates there  are  great  differences  in  individual  productivity. 
The  eight  most  "productive"  programmers  developed  239  hours  of 
instruction,  expending  a weighted  average  of  56  hours;  the  eight 
least  productive  programmers  developed  76  hours  of  instruction 
with  a weighted  average  expenditure  of  157  man-hours;  at  the 
extremes,  one  programmer  spent  1,389  man-hours  to  produce  one 
hoflr'Of  instruction  whlTe  another  programmer* produced  3^  in- 
structional hours  with  an  average  expenditure  of  31  man-hours 
per  Instructional  hour. 
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time  required  to  develop  instructional 
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Unknown  l?93'  l25>  (31E'  Poltyn.  Baudhum,  et  M.  1977 


Notes 


In  these  data,  there  is  no  evidence  of  differences  between 
students  and  project  staff,  and  there  is  nothing  in  the  infor- 
mation presented  to  tie  nan-hours  expended  with  difficulty  of 
either  the  material  programmed  or  the  instructional  approach. 

In  one  case,  two  student  programmers  were  employed  for  about 
**  years  and  produced  a total  of  97  hours  of  course  materials. 
Comparison  of  the  productivities  for  the  first  and  second  2-year 
periods  presents  evidence  of  appreciable  learning. 

The  values  of  man-hours  per  instructional  hour  given  by 
Hansen,  Ross,  Bowman,  and  Thurmond  (1975)  are  so  much  lower  than 
other  reports  that  they  are  immediately  suspect.  They  are  based 
on  the  three  courses  converted  to  CMI  at  the  Memphis  Naval  Air 
Training  Station  and  represent  roughly  300  hours  of  CMI  mate- 
rials. The  value  of  60  man-hours  per  instructional  hour  was 
derived  from  a survey  of  personnel  participating  in  thp  programs. 
However,  the  sample  was  extremely  limited;  of  13  individuals 
polled,  only  five  provided  quantitative  answers,  and  these  ranged 
from  10  to  150  man-hours  per  instructional  hour.  The  value  of 
30  man-hours  per  instructional  hour  is  presented  as  a "currently 
estimated"  requirement  including  "textual  media  conversion  as 
well  as  computer  activity"  but  does  not  reference  the  author 
survey  or  another  source. 

E . 5 PROGRAM  DELIVERY:  INSTRUCTION,  INCLUDING  INSTRUCTORS  AND 

INSTRUCTIONAL  SUPPORT  PERSONNEL  AND  STUDENT  PERSONNEL 

Studies  of  computer-based  training  typically  define  de- 
creases in  course  durations  as  an  element  of  training  effective- 
ness. However,  time  spent  by  military  personnel  In  any  function 
is  a cost,  since  personnel  received  resource  support  in  the  form 
of  salaries,  housing,  eta . Of  the  several  studies  that  addressed 
both  training  time  and  training  cost,  only  four  attributed  costs 
to  student  and  instructor  time. 

, . ..  . « w - 4 '*  ' 

Many  of  the  studies  reported  savings  due  to  reduced  times 
required  by  students  to  complete  courses  with  computer-based 
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instruction.  Since  computer-based  Instruction  also  requires 
some  additional  resources,  such  as  for  the  computer,  it  is 
obvious  that  it  is  not  clear  that  there  are  overall  savings  un- 
less both  values  are  expressed  in  comparable  terms,  i.e.,  dollars. 

The  omission  in  translating  student  and  instructor  time  to 
cost  is  inconsistent  with  the  treatment  generally  afforded  to  the 
cost  of  developing  instructional  materials.  In  most  cases,  the 
courseware  was  developed  by  military  personnel,  and  their  costs 
were  attributed  to  the  cost  of  the  courseware.  There  are  no 
grounds  for  distinguishing  these  personnel  costs  from  other  per- 
sonnel costs  that  were  not  included,  e.g .,  instructors  and  stu- 
dents . 

The  cost  information  developed  in  three  of  the  four  studies 
is  shown  in  Table  E-5-  The  $61,000  pay  and  allowance  rate  shown 
for  Crawford,  Hurlock,  et  al.  , (1976)  is  described  as  the  "billet 
cost  for  the  lowest  ranked  student  or  instructor"  (an  aviation 
lieutenant).  It  is  over  two  times  the  standard  pay  and  allow- 
ance factor  associated  with  junior  flying  officers  and  includes 
a variety  of  personnel  support  items  over  and  above  those  in- 
cluded in  pay  and  allowances,  e.g.,  command  and  administration, 
medical  costs,  dependents'  schools,  travel,  and  retirement. 

The  full  effect  of  computer-based  instruction  on  personnel 
costs  includes  its  impact  on  requirements  for  instructors  and 
other  types  of  direct  support  personnel,  and  computer-based 
instruction  is  generally  attributed  with  allowing  increases  in 
student : instructor  ratios.  From  the  information  shown  in  Table 
E-5,  this  increase  does  not  appear  to  have  a relatively  signifi- 
cant effect  on  cost.  Development  of  Instructional  materials  is 
a significant  cost  associated  with  the  introduction  of  computer- 
based  instruction,  and  the  extent  to  which  changes  in  student: 
instructor  ratios  offset  its  cost  depends  upon  the  number  of 
students  receiving  a given  hour  of  instruction.  Based  on  the 
sketchy  information  provided  by  Table  E-5  (assuming  an  expenditure 
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TABLE  E-5.  INSTRUCTOR  AND  SUPPORT  COSTS  FOR  PROGRAM 
DELIVERY,  COMPUTER-BASED  INSTRUCTION 


1 


DATA  ITEM 

SOURCE 

■ 

Hansen, 

Ross, 

Bowman,  & 

Thurmond 

(1975) 

Crawlord, 
Hurtock, 
Padillo,  & 
Sassano 
(1976) 

System 

Navy  CMI 

Navy  CMI 

PLATO  IV 

Change  in  Ratios 

Students  : Instructors 

10:1  to 

7.5:1  to 

a 

16.1 

9.0:1 

Students  to  Instructional 

- 

Unchanged 

- 

(Oirect)  Support 

at  24:1 

Students  to  Indirect 

•• 

Unchanged 

(Base)  Support6 

at  12.5:1 

Pay  and  Allowance  Rates 

wmm 

■ 

Students 

$5,899 

1 

Instructors 

9,697 

HGBSk 

1 

Instructional  Support 

•• 

•• 

llg^^ 

Indirect  Support 

- 

12.400 

’Cannot  be  expressed  m these  terms  The  net  result  was  to  eliminate  the  single  instructor  hour  contained  in  a 9-hour 
training  segment. 

6 Applies  to  students,  instructors,  and  instructional  support  personnel 

cThe  $61  000  figure  is  described  as  billet  cost  and  includes  a variety  ol  personnel  support  Hems  ever  and  above 
pay  and  allowances.  $g,  command  and  administration,  dependent  school  costs,  lecruiling  costs,  reenhslment 
bonuses,  and  retirement,  students  and  instructors  were  pilots. 
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of  250  man-hours  to  develop  a course-hour  and  a decrease  In 
course  duration  of  30  percent)  the  course  development  man-hours 
are  offset  by  a corresponding  reduction  In  Instructor  man-hours 
when  the  course  has  been  taught  to  approximately  3,100  students. 

By  contrast,  when  the  cost  avoidance  Is  based  on  the  combined 
decrease  in  instructor  and  student  costs  (assuming  the  pay  and 
allowance  rate  of  instructors  is  twice  that  of  students),  the 
breakeven  point  is  approximately  850  students.  Direct  support 
personnel  are  also  identified  with  program  management,  the 
operation  and  maintenance  of  laboratory  equipments  and  media 
devices,  and  other  instructional  support  functions,  but  no  infor- 
mation is  available  concerning  how  these  personnel  requirements 
are  affected  by  the  introduction  of  computer-based  instruction. 

The  fourth  study  that  treated  student  time  as  a cost  of 
training  (Polcyn,  Baudhuin,  et  at.,  1977)  investigated  the  use 
of  CMI  for  training  at  duty  stations  instead  of  at  training 
centers.  Schools  provide  training  at  both  initial  and  advanced 
skill  levels.  Both  require  transfer  of  students  to  schools  that 
result  in  expenditures  that  are  a cost  of  the  training.  Initial 
and  advanced  training,  occurring  between  duty  station  assignments, 
involve  an  additional  permanent  change  of  station  (PCS)  transfer 
resulting  in  costs  for  one-way  travel  and  movement  of  household 
goods.  Advanced  training  that  occurs  during  a duty  station 
assignment  involves  a temporary  duty  (TDY)  transfer  and  Incurs 
costs  for  round-trip  travel  and  per  diem  payments  for  the  dura- 
tion of  training. 

Data  were  presented  that  permit  estimation  of  transfer 
costs  (including  per  diem)  per  course:  $425  for  advanced  train- 
ing based  on  PCS  transfer,  $400  for  advanced  training  based  on 
TDY  transfer,  and  $140  for  initial  training  based  on  PCS  trans- 
fer. A significantly  higher  cost  ($825)  for  PCS  transfer  for 
advanced  training  is  cited  by  the  Air  Force  in  "USAF  Cost  and 
Planning  Factors"  (Air  Force  Regulation  173-10).  To  the  extent 
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that  CMI  and  CAI  permit  training  to  be  provided  at  duty  stations, 
the  resulting  decrease  in  relocation  costs  Is  a true  cost  avoid- 
ance . 

E . 6 PROGRAM  DELIVERY:  LABORATORY  EQUIPMENT 

No  data  were  found  on  these  costs  of  instruction. 

E . 7 PROGRAM  DELIVERY:  MEDIA  DEVICES 

The  cost  of  using  media  devices  presents  methodological 
problems  that  were  not  addressed  in  the  literature  reviewed. 

Media  devices  are  long-lived,  independent  of  subject  matter,  and 
generally  portable;  these  attributes  introduce  some  problems  in 
determining  the  costs  of  their  use  in  a course  or  course  segment. 

Unit  procurement  cost  data  are  easy  to  compile  from  such 
sources  as  published  catalogues.  Table  E-6  contains  unit  cost 
and  other  data  for  broad  groupings  of  equipments.  The  wide 
range  of  costs  is  a function  of  device  size  and  other  features. 
Note  that  the  range  of  costs  shown  encompasses  an  order  of  mag- 
nitude yet  does  not  include  devices  sized  for  presentation  to 
large  groups,  such  as  in  auditoriums.  Determining  their  costs 
of  use,  in  general,  requires  further  data  regarding  operating 
costs,  failure  rates,  and  repair  costs.  A second  citation  to 
failure  rates  is  shown  in  Table  E-7,  but  we  could  find  no  infor- 
mation on  repair  costs.  Their  full  cost  of  use  would  also  require 
data  regarding  lifespans  in  order  to  amortize  initial  costs,  and 
only  Hess  and  Kantar  (1977)  contained  any  information. 

The  methodological  problems  arise  in  determining  cost  of 
use  in  a particular  training  situation.  The  first  problem  in- 
volves selection  of  devices.  A variety  of  media  are  available 
for  presentation  of  instruction,  and  different  media  can  be  se- 
lected for  presenting  the  same  subject  matter  in  small  units  of 
Instruction,  e.g.,  individual  lessons.  As  a result,  a course 
(or  segment  of  a course)  may  utilize  an  extensive  mixture  of 
media.  With  the  variation  in  the  cost  of  media  devices  noted. 
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TABLE  E-6.  MEDIA  DEVICES:  UNIT  COST  RANGES 
AND  OTHER  INFORMATION® 


Media  Device 

Range  of 
Initial  Cost 
(Dollars) 

L i fe 
Span 
(Years ) 

MTBFC 
(Hours  ) 

Sound  Movie  Projectors 

$175  - 1000 

6 

90  - 1 10 

Videotape  Recorders/Players 

$600  - 8000 

5 

Sound  Slide/Strip  Projectors 

$100  - 1000 

Silent  Movie  Projectors 

$150  - 250 

Silent  SI i de/Stri p 

Pro jec  torsb 

$ 25  - 900 

6 - 10 

90  - 150 

Random  Access  Slide  Projectors 

$500  - 2000 

Microf i lm/Fiche  Readers 

$ 80  - 800 

Audiotape/Disc  Players 

$ 30  - 325 

Teaching  Machines  (Individual) 

Audio  Visual 

Rate  Control 

$230  - 1000 

Constant  Control 

$1950 

Visual 

Rate  Control 

$140  - 380 

Constant  Control 

$220  - 1200 

Audi  o 

Rate  Control 

$190  - 470 

Excludes  equipments  too  large  for  use  in  individual  classrooms.  Costs 
are  for  commercial  quality  equipments. 

includes  overhead  projectors. 

cMean  time  between  failures. 

Source:  Hess  and  Kantar,  1977. 
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TABLE  E-7.  MEDIA  DEVICE  RELIABILITY 


Media  Device 

Hours  of 
Use 

Number  of 
Fai lures 

MTBFd 
(Hours  ) 

Motion  Picture  Projector 

2,364 

12 

197 

Videotape  Player 

2,315 

5 

463 

TV  Monitor 

2,315 

2,315 

Sound/Filmstrip  Projector 

102,509 

125 

820 

Slide  Projector 

3,711 

3,711 

Silent  Filmstrip  Projector 

12,427 

33 

377 

Microfiche  Reader 

30,636 

1 1 

2,785 

Audio  Tape  Player 

33,394 

12 

2,783 

Headset 

136,199 

5 

27,240 

aMean  time  between  failures. 

Source:  AIS  Integrated  System  Test  (Draft).  McDonnell 
Douglas,  1977a. 

different  mixes  of  media  can  imply  sizeable  differences  in  course 
costs.  Determination  of  the  device  mix  implies  an  extensive 
course  design  effort,  including  specification  of  equipment-to- 
student  ratios  and  the  environments  in  which  different  equip- 
ments will  be  used  ( e.g . , individual  or  auditorium  presentation). 
The  systematic  investigation  of  alternative  course  designs  to 
determine  cost  differences  associated  with  these  mixes  could  be 
an  expensive  process. 

A second  problem  arises  from  the  physical  nature  of  media 
devices.  Even  if  a mix  of  media  devices  were  formulated  for  a 
training  course  or  course  segment,  the  cost  of  using  those  de- 
vices in  that  course  must  still  be  defined.  Media  devices  are 
long-lived  and  independent  of  subject  matter;  once  procured, 
they  serve  as  an  inventory  to  satisfy  requirements  levied  by  all 
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courses  offered  at  an  installation.  Operations  and  maintenance 
costs  may  be  based  on  individual  course  usage  in  a straight- 
forward manner.  However,  the  full  cost  of  ti.e  use  of  media  de- 
vices will  depend  upon  whether  or  not  existing  stocks  are  suffi- 
cient to  meet  the  demands  of  all  users.  If  stock  levels  are 
adequate,  equipments  will  be  available  for  proposed  courses  (the 
incremental  user)  with  no  additional  outlays;  if  not,  procurement 
of  additional  equipments  would  be  indicated  and  these  purchase 
costs  must  be  accounted  for  in  some  manner.  Information  regard- 
ing the  adequacy  of  existing  inventories  and  user  requirements 
is  rarely  available,  and  there  are  a number  of  w ays  in  which 
purchase  costs  can  be  apportioned.  Analyses  must  resort  to 
assumptions  and  allocation  schemes  that  are  essentially  arbitrary 
and  the  results  would  be  dependent  upon  .lust  what  assumptions  and 
schemes  were  employed. 

E . 8 INSTRUCTIONAL  PROGRAM  DELIVERY:  COMPUTER  SYSTEMS 

Information  on  instructional  computer  hardware  costs  is  un- 
expectedly hard  to  find  in  the  literature,  despite  the  central 
role  it  plays  in  system  capability  and  costs.  Computer  hardware 
appears  to  account  for  no  more  than  a modest  share  of  computer- 
based  instruction  system  life-cycle  costs.  However,  a purchase 
of  hardware  represents  the  bulk  of  early  system  cost  and  amounts 
to  a commitment  to  CBI  and  the  other  costs  and  risks  that  commit- 
ment entails. 

Substantive  information  was  found  on  five  hardware  systems — 
PLATO  IV,  TICCIT,  GETS,  IBM  1500,  and  Navy  CMI;  some  information 
is  available  on  a one-of-a-kind  configuration  assembled  to  eval- 
uate shipboard  use  of  CMI.  One  additional  source  contained  a 
single  aggregate  figure  for  development  of  the  Air  Force  Advanced 
Instructional  System  (AIS).  However,  this  figure  represented  a 
contract  value  for  a mixture  of  hardware,  software,  and  course- 
ware development  that  could  not  be  separated  by  function. 


E-19 


Taken  system  by  system,  the  detail  and  completeness  of  the 
data  are  diverse.  In  no  case  was  the  material  sufficient  for 
an  adequate  understanding  of  the  drivers  of  system  cost.  Indi- 
vidual studies  are  typically  limited  in  scope,  and  the  cost 
information  presented  is  generally  limited  to  only  that  necessary 
to  the  principal  issue  addressed.  For  example,  in  experimental 
programs,  the  costs  that  were  reported  were  normally  limited  to 
those  directly  (and  incrementally)  incurred  as  a result  of  the 
program,  i.e.,  those  that  could  be  identified  as  budget  expendi- 
tures of  the  demonstration. 

The  discussion  that  follows  is  organized  according  to  com- 
puter system  rather  than  type  or  location  of  hardware.  Note 
that  all  costs  are  displayed  as  then-year  dollars  rather  than 
being  adjusted  to  a common  base  period.  In  the  light  of  chang- 
ing CAI/CMI  hardware  configuratios  and  technology,  no  satisfac- 
tory price  index  could  be  found.  In  several  studies,  it  appeared 
that  costs  given  were  not  current  with  the  publication  date,  and 
no  information  was  provided  as  to  the  applicable  dates  of  the 
costs . 

E.8.1  PLATO  IV. 

More  studies  have  addressed  PLATO  IV  than  any  other  system. 
Although  purchase  of  a PLATO  IV  system  is  very  expensive,  access 
to  its  services  can  be  purchased  in  small  and  divisible  units, 
e.g.,  the  individual  terminal,  witn  a small  initial  outlay,  and 
its  software  system  is  highly  developed.  These  features  are 
particularly  attractive  for  small-scale  programs,  typical  of 
experimental  applications. 

Most  military  use  of  PLATO  IV  was  supported  by  a contract 
between  DARPA  and  the  military  Services  with  the  Computer-Based 
Education  Research  Laboratory  (CERL)  at  the  University  of 
Illinois.  This  arrangement,  however,  casts  some  question  on  the 
value  of  cost  data  reported.  In  addition  to  being  non-profit, 
CEP.L  is  the  originator  of  PLATO  IV  and  has  a vested  interest 
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in  its  use.  The  CERL  contract  charges  are  considerably  less 
than  those  of  Control  Data  Corporation  (CDC),  currently  provid- 
ing PLATO  IV  services  on  a commercial  basis;  the  CERI.  charges 
may  well  be  less  than  the  military  could  provide  the  same 
services  for  on  an  in-house  basis.  The  values  shown  in  Table 
E-8  are  reported  actuals  from  experimental  programs  funded 
through  the  CERL  contract  for  Hurlock  and  Slough,  1976;  U.S. 

Army  Ordnance  Center  and  School,  1975,  Dallman,  DeLeo  et  at., 
1977;  and  Steinkerchner , Deignan,  et  al . , 1977.  The  values 
shown  for  Kribs,  1976,  for  terminals  and  PLATO  IV  system  support 
and  the  system  support  shown  for  Crawford,  Hurlock,  et  al.,  1976, 
are  quotations  from  CDC. 

The  costs  displayed  in  Table  E-8  are  based  on  the  use  of  a 
small  fraction  of  the  central  hardware  capability  of  a PLATO  IV 
system.  The  computer  access  is  charged  to  cover  central  system 
operating  costs,  amortization  of  the  hardware,  and  other  services 
provided  by  CERL  on  the  basis  of  the  number  of  terminals  con- 
nected. Large-scale  use  of  PLATO  IV  by  the  Services  would  prob- 
ably involve  purchase  of  both  terminal  and  central  processor 
hardware  with  a sizeable  Initial  outlay. 

Control  Data  Corporation  has  provided  an  estimate  of  hard- 
ware procurement  costs  for  a 1000-terminal  system  capable  of 
supporting  725  terminals  simultaneously  in  a student  mode,  or 
*425  in  an  author  mode.  The  configuration  and  component  costs 
are  shown  in  Table  E-9»  The  simultaneous  terminal  constraint  is 
a function  of  central  processor  cycle  time  rather  than  the  "swap" 
time  between  the  random  access  and  swap  memories.  Saturation  of 
a swap  memory  capacity  is  reached  at  approximately  800  students; 
this  level  restricts  student  blocks  to  14000  words  when  all  stu- 
dents are  accessing  different  programs. 

As  shown  in  Table  E-9,  the  configuration  contains  components 
not  currently  available  on  a commercial  basis.  The  extended  swap 
memory  (ESM),  currently  in  development,  is  of  metal  oxide  semi- 
conductor (MOS)  technology  and  will  employ  a controller  with 
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TABLE  E-8.  COSTS  ASSOCIATED  WITH  THE  USE  OF  PLATO  IV 


Item 


Numbe*  ol  Termif*ls 
On  line 


See  Note 


PlAlO  IV  Terminal 
Basic 

Touch  Panel 
Slide  Option 
Audio  Unit 


PLATO  IV  Srslem  Support 
Computer  Access  Fee 

Maintenance  and 
Other 


Hurt**  end 
Slough  1976 


I? 

~~r 


9 | Code 


1Q.SOO  | P.T 

1 . 


2 MO  I TC  i 

I 

I 

1.300  I T.C.t 


Communications 

1 1 

Commercial  ! 

i 

j 

Lease  lme 

i 

GSA  lease  Lme 

50  | M.4T 

Microwave  System  ] 

1 

| 

j 

Multiplexor 

10  500  P.4T 

i 

Da'a  Pori 

1 

1 

Van an  Punier 


— 1 

Crawford.  I U S Aim*  Ordnanco 
Hurtoch.  | Cantoi  and  Softool 

ti  ol . 1976  ! 197S 


Oodman.  Ooloo. 
oi  ol  1977 


j Slorftlioichnor. 
Doitnoft. 
1977 


10.900 


Cod* 


P.T 

I 


35  820  0 41 


Kite.  1926 


24 


Codo 


Cod*  I « | Codo  I 4 


2.500  I T C T 


1.000 


1C  1 


204 


9.630 


Code 


5.205 

P,T 

| P-T  i 

I X.P.T 

i 

544 

P.T 

! PT  1 

1 i 

1 

600 

| PT 

1 

1 

2.00C 

P.T 

PT 

1500 

TC  t 

1500 

IC 

7? 00 

1 

c.x.T.r 

T C v 

_ 1 

TC.r 

1 

i960 

l 1 

C.x.T.r 

1 

I 7.686 


M Per  r lie  month 

T Per  lei  now 

41  Per  four  terminals 

V Per  year 


Code  C Contract  service  or  lease 
P Pc  chase 
X (iuolaticn 

I Included  m basic ' cost 

\ofcs 

1 These  studies  report  the  results  of  small  scale  evaluations  ol  the  PLATO  IV  system  that  were  partially  funded  Lf  CAR?*  Since  the  programs 
*?re  complete  ai  the  time  of  documentation  all  costs  show  are  assumed  to  be  actuals  except  those  noted  as  esimates  DARPA  suppor 
included  all  items  shown  in  the  laple  and  me  would  eipec‘  to  bryl  the  tosls  of  like  items  to  he  the  same  The  higher  maintenance  cost  noted 
.n  Hurtotk  and  Slough  1976  is  eipiamed  hy  the  longer  distance  between  S*n  Oifeyo  nnd  the  University  ril  Illinois,  hut  the  documentation  provided 
no  explanation  lor  the  other  differences 

? The  ctpt omcntai  pruyram  addressed  m uiawfu'd  nu'ioc*  el  «•  *376.  *s  on*;  ‘it  a 'oii'lit'  <|I  pi  lytams  addressed  m Huriock  and  Stougfi  5376 
The  lower  PLATO  IV  system  support  costs  shown  by  Huriock  and  Slough  1976,  is  tiaseo  on  the  CERl  contract  while  the  higher  value  shown  in 
Crawford  Huriock.  er  a/  1976  is  a Quotation  solicited  from  COC  The  estimates  shown  in  Knbs  1976  for  holt,  the  terminal  and  system  support 
were  obtained  from  COC 

j Communications  for  this  program  were  achieved  hy  microwave  lor  'ransmission  from  CERl  to  Chanult  AFB  <a  distance  of  IS  miles*  and  local 
telephone  kites  lor  transmission  from  Changfe  to  CEPl 

4 The  has«c  terminal  is  noted  as  including  a local  MODEM 
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TABLE  E-9.  PLATO  IV  HARDWARE  PROCUREMENT 
AND  MAINTENANCE  COSTS 


Model 
Numbe  r 

fear  of 
Availa- 
bility 

Quant i ty 

1 

Pure  nase 
Price 

v 000 ) 

| Maintenance 
Cost 

(COO) 

i 

See 

Note  | 

Central  Site  Equipment 

T 

j 

j 

I 

Central  Processor 

1978 

) 

1 

31  529 

i 3 4.6 

• 1 

Extended  Swap  Memory 

“ " 

1981 

l 

i 

1351 

! 

J ; 

Computer  Interface  Unit 

4001 

H>8 

1 

14 

0 . 1 

3 ! 

Computer  Interface  Unit  Expansion 

4001  -2 

1 9 7 A 

2 

i 

1 2 

1 0.0 

4 j 

Tape  Controller 

721 *21 

1978 

2 

61 

, 0 . 3 

1 

Tape  Drive 

667.2 

1978 

i 

4 1 

i 0.3 

U i S k Con  t rol 1 e r 

7054-41 

1978 

4 

297 

; .C 

i 

D i Sk  Drive 

844-41 

1 97S 

20 

1 

^15 

1 2.1 

i 

Line  Printer 

680-12 

1973 

1 

! 

71 

u.  3 

; 

S'te  Controller 

40002-1 

1978 

22 

i 

213 

2.8 

5 

Remote  Interface  Module 

40002-  13 

1 978 

1 76 

i 

211 

• 0.9 

6 

Mul  ttpUior.'HOOCH 

-• 

19/9 

125 

686 

3 . 1 

! 

Total 

i 

55003 

! 517.3 

Remote  Site  Equipment 

1 

Terminal 

1 ST  2 

I960 

1000 

i 

35000 

1 325.0 

8 ^ 

Multiplexor /MODEM 

1 9 7 9 

125 

i 

6Sd 

1 3'' 

7 | 

Total 

i 

_L 

55688 

1 

528.: 

Source 
Notes • 


Unpublished  materials  received  from  and  conversations  R . A.  Moe  of  Control  Data  Corporation. 


Ms  after  the  first  2S6  terminals 


1.  C:BER  174-6  with  131*  words  of  random  access  memory. 

?.  MOS  technology  with  4-iPllllon  word  capacity. 

3.  Capable  of  controlling  256  active  terminals.  ^ 

4.  Eacn  e*oanston  unit  is  capable  of  controlling  2S6  active  termi 
controlled  by  the  master  computer  interface  unit. 

5.  Capable  of  controlling  32  active  terminals 

6.  Capable  of  controlling  4 active  terminals  (i 

7.  Eight-port  multiplexor  and  9n00  bps  MODEM.  ^ 

8.  Production  version  of  the  current  terminal.  Prlncloal  changes  to  permit  production  of  larger  lot 
sizes.  The  number  of  circuit  boards  will  be  reduced,  the  keyboard  made  integral  to  the  case,  and 
tube  will  be  sma 1 1 er  . 


eight  per  site  control ler) . 
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eight  times  the  addressing  capability  of  those  used  with  the 
current  magnetic  core  units.  The  higher  capability  controller 
will  permit  ESN!  growth  to  16  million  words,  compared  with  the 
2-mlllion  word  limit  of  the  current  controller.  The  terminal  is 
a smaller  integrated  unit  with  emphasis  placed  on  producibility 
of  larger  lot  sizes.  The  eight-port  multiplexor  is  currently  in 
pilot  use.  Its  impact  on  system  costs  lies  in  the  potential  of 
reducing  the  number  of  multiplexors  and  MODF.Ms  and  the  costs  of 
communications  services  by  up  to  ono-half.  With  one  exception, 
the  prices  shown  are  current  values  rather  than  projected  values. 
The  one  exception  is  the  extended  swap  memory  where  the  recent 
trend  decline  in  KOS  chip  costs  has  been  projected  to  reflect 
anticipated  1981  costs. 

E.8.2  TICCIT. 

Very  little  information  was  four.d  concerning  TICCIT  hard- 
ware costs  in  the  literature.  Much  of  what  was  found  is  out  of 
date  as  the  result  of  a recent  major  configuration  change  insti- 
tuted by  Hazeltine  Corporation,  which  currently  markets  TICCIT 
on  a commercial  basis.  Among  other  changes,  the  new  configura- 
tion substitutes  the  Nova  3/0  central  processor  for  the  Nova 
800  processor. 

One  description  of  TICCIT  hardware  and  costs  Is  shown  in 
Table  E-10,  (Kearsley,  1977).  This  information  was  obtained 
from  the  MITRE  Corporation  in  197^  and  pertains  to  the  Nova  800 
configuration.  Kribs  ( 1976)  provides  an  estimate  of  $270, 0C0 
for  hardware  and  $130,000  for  system  installation  and  checkout 
for  a 13-terminal  configuration.  This  figure  is  in  close  agree- 
ment with  the  MITRE  Information  and  is  based  on  the  NOVA  800 
configuration  installed  at  North  Island  Naval  Air  Station. 

Unpublished  Information  received  from  Hazeltine  regarding 
the  new  configuration  indicates  that  the  central  system  has  a 
degree  of  modularity  and  may  consist  of  one  or  two  central  pro- 
cessors, depending  upon  the  number  of  terminals  to  be  supported, 


E-24 


TABLE  £-10.  ESTIMATED  COST  OF  TICCIT  HARDWARE 
(The  Mitre  Corporation,  1974) 


Central  Processor  Unit 

Main  Processor 

$34,000 

Terminal  Processor 

15,000 

Disc  Drive  & Controller 

(3  at  $23,300 

each ) 

70,000 

Tape  Unit 

9,000 

Card  Reader 

4,000 

Line  Printer 

11 ,000 

CRT  Terminal 

3,000 

Computer-Computer  Line 

3,000 

Character  Generator 

7,000 

Keyboard  Interface 

6,000 

Audio  System 

56,000 

Refresher  Control 

6,000 

Crossbar  Switch  (Audio/Video  Switching) 

17,000 

Video  Tape  Players  (20 

at  850  each) 

17,000 

Cabinets 

7,500 

Total 

$265,500 

Terminals 

TV  Monitor  (including 
modi fi cation ) 

128  at 

$320 

Keyboards 

128  at 

$170 

Refreshers 

128  at 

$735 

Signal  Processors 

128  at 

Total 

$125 

$TT5tj 

$172,800 

Total 

$438,300 

and  varying  numbers  of  disc  units , depending  upon  the  extent  of 
courseware  and  number  of  students  to  be  supported.  Hazeltlne  has 
provided  what  is  described  as  "budgetary  pricing  information"  as 
of  June  1977,  as  shown  in  Table  E-ll.  Hazeltine  indicates  that 
as  of  June  1978  these  prices  had  increased  approximately  20  per- 
cent, bringing  the  hardware  cost  of  a 128-terminal  system  to 
well  over  $1  million. 

E.8.3  IBM  1500 

Three  references  to  IBM  1500  system  costs  were  found  in  the 
literature,  and  one  included  the  IBM  1M60  system,  as  summarized 
in  Table  E-12.  Two  sources  provided  tabular  breakouts  (Kopstein 
and  Seidel,  1969  and  Kearsley,  1977)  while  the  third  (Ford, 

Slough,  and  Hurlock,  1972)  provided  a single  "bottom  line"  cost. 
Only  Kearsley  (1977)  explicitly  stated  that  the  costs  came  from 
accounting  records  of  operating  experience. 

Two  points  are  to  be  noted  regarding  this  material.  The 
first  is  that  the  hardware  costs  are  monthly  lease  values  in  all 
cases,  with  maintenance  provided  by  the  contractor.  The  second 
is  that  the  lease  costs  are  roughly  the  same,  although  the  publi- 
cation dates  span  a 10-year  period  of  rising  general  cost  levels 
and  falling  costs  of  computer  service.  The  discussions  in  the 
references  were  insufficient  to  explain  the  similarity)  but  some 
rationale  may  be  gleaned  from  the  different  environments  on 
which  the  estimates  are  based.  The  values  provided  by  Kearsley 
(1977)  are  based  on  an  extended  period  of  operation  that  began 
in  the  late  19o0s;  it  was  not  noted  to  which  part  of  the  time 
period  the  values  pertain.  Further,  the  system  was  installed  at 
a Canadian  university  where  relative  costs  may  differ  from  those 
in  the  United  States.  Data  provided  by  Kopstein  and  Seidel  (1969) 
are  based  on  civilian  installations,  while  those  of  Ford,  Slough, 
and  Hurlock  (1972)  are  based  on  military  use.  With  the  different 
ways  data  categories  are  combined,  it  is  possible  that  central 
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TABLE  E-ll.  ESTIMATED  COST  OF  TICCIT  SYSTEM 
(Hazel tine  Corporation,  1977) 


Basic  Systems 

System  I (32  Terminals)  Capacity 

System  II  (64  Terminals)  Capacity 
System  III  (128  Terminals)  Capacity 

$520,000 

585.000 

630.000 

Options 

Graphic  Digitizer 

25,000 

Disc  Drive 

21 ,000 

Software  Support  Package 

(Not  available  for  System  I) 

24,000 

Optical  Spares  Package 

21,000 

Video  Tape  Option  - System  I 

45,000* 

System  I I 

60,000* 

System  III 

90,000* 

Audio  Option  - System  I 

100,000 

System  II 

100,000 

System  III 

70,000 

Terminal:  Average  Cost  for  Quantity: 

32 

2,375** 

64 

2,325** 

128 

2,275** 

♦Deduct  $20,000  when  audio  option  is  also  included. 
♦♦Add  $250  for  light  pen. 


Source:  Hazeltine  Corporation,  1978. 
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TABLE  E-12 . IBM  1500  SYSTEM  COSTS 


Includes  Maintenance. 


processor  lease  rates  might  have  declined  and  maintenance  costs 
increased  over  time  and  without  this  being  noticeable. 

£.8.4  Navy  CM1  System . 

The  Navy  CMI  system  has  used  two  different  central  hardware 
systems.  Initial  development  and  application  of  the  system  was 
accomplished  using  a Xerox  Sigma  9 system,  located  at  Memphis 
State  University  and  dedicated  to  Navy  use  during  regular  train- 
ing hours.  Later,  the  Navy  procured  a Honeywell  Series  60  Level 
66  system.  The  configuration,  shown  in  Table  F.-13,  la  described 
in  Hansen,  Ross,  et  al . , 1975.  At  the  time  of  publication  the 
hardware  was  under  contract  for  procurement,  and  we  have  not 
verified  whether  configuration  changes  occurred  between  then  and 
its  final  installation.  Costs  of  its  components  were  obtained 
from  issues  of  computer  price  survey  publications,  as  noted. 

Student  terminal  configuration  and  cost  information  is 
taken  from  Poleyn,  Baudhuln,  et  al.  , (1977)  and  is  shown  in 
Table  E-1U.  The  date  of  this  publication  would  allow  these 
values  to  be  based  on  historical  records  that  antedate  Installa- 
tion of  the  Honeywell  central  processor.  With  regard  to  main- 
tenance costs,  neither  the  philosophy  (in-house  or  contract)  nor 
a breakdown  between  labor  and  material  was  given, 

E.8.5  General  Electric  Training  System  (GETS). 

GETS  (aloo  known  as  Computer-Based  Training  System  or  CBTS) 
is  a stand-alone  (single  terminal)  unit.  The  only  military  appli- 
cation at  present  is  for  crew  training  for  the  TPIDENT  system 
when  there  are  more  students  than  the  regular  laboratory  equip- 
ment can  handle.  A life-cycle  cost  evaluation  (Kribs,  1976) 
estimated  unit  hardware  production  cost  to  be  $3^,000  and  other 
nonrecurring  costa  associated  with  procurement  of  13  units 
(spares,  manuals,  installation,  etc.)  at  approximately  $125,000. 
These  values  are  based  on  a contractor  quote  that  is  several 
years  old,  and  its  timeliness  is  open  to  serious  question. 
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TABLE  E-13.  NAVY  CMI  SYSTEM:  INITIAL  CENTRAL  SITE  HARDWARE  COST 


oney we 


TABLE  E-14.  NAVY  CMI  STUDENT  TERMINAL  COST 


Item 

Un  i t 
Initial 
Cost 

Un  i t 

Annual 

Maintenance 

Opscan  17,  Optical  Scanner 

$ 8,998 

$ 82.8 

Opscan  Auto-Feed 

652 

60 

GDC -202-9D  Modem 

400 

60 

Terminet  1 200,  Key  boa rd/ P r i n te r 

4,200 

816 

Total 

$14,250 

$1 ,764 

Source:  Polcyn,  Baudhuin,  et  al.,  1977. 


Technological  advance  in  the  intervening  period  has  had  its 
greatest  impact  on  the  types  of  components  used  in  stand-alone 
systems;  in  particular  the  costs  of  microprocessors  and  both 
integral  and  peripheral  data  storage  devices  have  experienced 
relative  declines. 


E.8.6  Sh  ; pboard  Computer-Based  Instruction. 


Shipboard  use  of  automated  data  processing  for  non-tactical 
applications  has  been  investigated  by  the  Navy  for  about  10 
years.  Two  areas  of  application  have  been  investigated  under 
the  designation  "Automated  Shipboard  Information  Management 
System  (ASIMS)".  The  first  is  a general  data  management  capa- 
bility "Command  Management  System  (CMS)",  and  the  second  is 
computer-based  instruction,  "Computer  Integrated  Instruction 
(CII)".  The  first  shipboard  test,  a Data  General  NOVA  1200 
system  installed  aboard  the  USS  Dahlgren  in  1973,  emphasized 
Command  Management  System  applications.  In  1975,  the  system  was 
transferred  to  the  USS  Grldley  where  emphasis  shifted  to  Com- 
puter Integrated  Instruction  applications.  The  NOVA  1200  has 
now  been  replaced  with  a Digital  Equipment  Corporation  (DEC) 
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POP  11/60,  and  the  application  has  been  extended  to  include  both 
cris  and  CII. 

Both  the  NOVA  1200  and  PDP  11/60  arc  considered  mini-com- 
puters. Little  information  was  found  concerning  the  configura- 
tion or  cost  of  this  installation.  The  configuration  displayed 
in  Table  E-15  is  contained  in  unpublished  information  received 
from  the  Navy  Personnel  Research  and  Development  Center  (undated) 
and  pertains  to  the  USS  Gridley.  Component  costs  were  contained 
in  a discussion  of  the  USS  Dahlgren  installation  in  Middleton, 
Papetti,  and  Michel!  (197*0.  There  is  an  ambiguity  between 
these  two  sources  concerning  whether  the  costs  shown  for  the 
disc  drives  and  CRTs  represent  unit  costs  or  the  costs  of  two 
and  four  units,  respectively.  In  Table  E-15,  they  are  treated 
as  costs  of  the  quantities  shown;  If  they  should  have  been 
treated  as  unit  costs  the  total  Installation  cost  would  rise  to 
over  $100,000.  Note  that  the  cost  values  are  based  on  1973  in- 
formation, and  it  is  questionable  whether  they  represent  the 
current  costs  of  mini-computers  of  like  capability. 

TABLE  E-15.  HYPOTHETICAL  SHIPBOARD  SYSTEM:  ESTIMATED 
HARDWARE  PROCUREMENT  COST 


r 


Component 


Ce 

n t r 

al  Proc 

es  so 

r (NOVA 

1 200  , 

3 2 K 

Memory ) 

Li 

ne 

Printer 

and 

Con  t ro 

1 1 e r 

( 

365 

1 pm ) 

Ca 

rd 

Reader 

and 

Control 

1 er 

(1 

00  c 

pm) 

Di 

SC 

Drive  a 

n d C 

on  tro  1 1 

er 

(2 

units, 

1 2 m 

i 1 1 i 0 n 

16-b 

i t 

wo  r 

d s each) 

A 1 

pha 

name r i c 

CRT 

(4  uni 

ts  , 

Ha 

ze  1 1 

ine  2000) 

Ot 

her 

(Casse 

tte 

Tape  Dr 

i ve  s 

* 

Tele 

type ) 

$20,500 
13,100 
5 


24,000 


$68,650 


Source:  Unpublished  information  received  from  the  Naval  Personnel 
Research  and  Development  Center  (undated);  also 
Middleton,  Papetti,  and  Michel i,  1974. 
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E . 8 . 7 System  Cost  Comparison  . 


The  hardware  systems  discussed  in  this  Appendix  include  the 
extremes  of  current  system  capabilities  in  terms  of  the  number 
of  terminals  supported.  The  cost  of  these  systems  can  be  ex- 
pressed in  three  ways:  (1)  system  procurement  cost,  (2)  system 
procurement  cost  per  terminal,  and  (3)  system  procurement  cost 
per  student-hour  based  on  assumed  life  spans  and  utilization 
rates.  Table  E-16  displays  the  comparisons  between  each  system 
for  each  of  the  three  measures.  The  inverse  relationship  between 
system  size  (or  Initial  procurement  cost)  and  cost  per  connected 
terminal  or  per  student  hour  is  rather  consistent  within  the  data 
sample,  but  we  have  strong  reservations  regarding  the  reliability 
and  timeliness  of  the  data  and  of  drawing  strong  conclusions 
based  on  them.  Although  it  appears  safe  to  say  that  computer 
hardware  for  the  Navy  CMI  Instructional  system  costs  less  per 
student  hour  than  the  computer  hardware  for  any  of  the  CAI  in- 
structional systems,  it  should  also  be  obvious  that  other  computer 
system  costs,  such  as  for  installation  and  maintenance,  are  not 
included  in  these  values. 

E . 9 PROGRAM  DELIVERY:  COMMUNICATIONS 

Communications  costs  arise  only  in  large  computer-based 
systems  where  terminals  may  be  located  at  appreciable  distances 
from  central  processors.  Currently,  this  applies  only  to  PLATO 
IV  and  the  Navy  CMI  systems . Four  methods  of  communication  have 
been  suggestea--land  lines,  microwave  systems,  satellites,  and 
closed-circuit  television.  All  applications  of  these  systems, 
with  the  exception  of  one  PLATO  IV  experimental  program,  have 
employed  land  lines.  The  program  pursued  at  Chanute  AFB,  re- 
ported in  Dallman,  DeLeo,  et  al . , 1977,  utilized  a microwave 
system  as  a primary  communication  link. 

The  PLATO  IV  programs  that  utilized  land  lines  employed  GSA 
leased  lines  with  costs  of  roughly  $.50  per  mile  per  month.  This 
rate  was  also  cited  in  several  other  studies  where  it  was  also 
reported  to  be  approximately  one-half  of  the  commercial  interstate 
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TABLE  E-16.  COSTS  OF  COMPUTER  SYSTEMS  HARDWARE 


Method 

ol 

Inilruction 

Computer  Syiiem 

Central 

Processor 

Cost 

(Thousands) 

Terminal, 

Unit  Cost 
(Thousands) 

System 

Hardware 

Cost 

(Thousands) 

Syttem 

Hardware  Cost 

Per  Terminal 
(Thousands! 

System 
Hardwire  Cost 

Per  Stvdem 

Hour* 

IBM  1500 

32  Terminals* 

_ 

_ 

S 800 

S 75 

S 2 49 

PLATO  IV 

1.000  Termin»lsc 

s s.ooo 

S 5.7 

10.700 

11 

1 48d 

CAI 

TICCIT 

32  Termin*lie 

760 

2 9 

8S0 

27 

2 66 

64  Terminals 

870 

2.8 

1.050 

16 

1 64 

126  Terminals 

970 

2.8 

1.330 

10 

1 04 

GETS 

One  Terminal 

- 

' 

34 

34 

3 40 

Navy  CMI 

6.000  Student!* 

2.300 

14  3 

4,020 

34 

007 

CMI 

16.000  Students* 

2.300 



14  3 

6,160 

22 

0 04 

4 IB  79 

*7.000  fceu't  per  lermimi  per  year  for  S yean 

^Includes  maintenance  Ba«eP  *n  teeie  ralei  and  amortizing  equipment  over  a 5 year  pened.  1967.  197?.  1977 
^Control  OaH  Corporation  quotation  from  private  communication  Plied  14  Auguit  1970 
tf8ated  on  776  active  terminal  constraint 
*Haitlline  quotation,  from  private  communication.  1976 
170  terminal!  at  SO  HuPenlt  per  terminal  1977 
0370  ferminatt  at  SO  ituPenti  per  terminal.  197  7 
3 76  79  3S 


rates.  In  fact,  the  commercial  tariffs  are  based  on  complex  rate 
structures  set  by  the  Federal  Communications  Commission  for  Inter- 
state (or  long)  lines  and  individual  states  for  wholly  intrastate 
(or  short)  lines. 

The  rate  schedule  (Table  E-17)  for  commercial  long  lines  was 
found  in  Ball  and  Jamison,  (1973)  and  Middleton,  Papetti,  and 
Micheli  (197*0: 

TABLE  E-17.  RATE  SCHEDULE  FOR  COMMUNICATIONS 
OVER  COMMERCIAL  LONG  LINES 


INSTANCE 

INTERVAL 

(MILES) 

COST  PER  INCREMENTAL 

MILE  PER  MONTH 

AVERA6E  COST  PER  MILE 
PER  MONTH  (AT  LIMIT  OF 
DISTANCE  INTERVAL 

1-25 

$3.30 

$3.30 

26-100 

2.31 

2.56 

101-250 

1.65 

2.01 

251-500 

1.15 

1.58 

>500 

0.83 

1.20* 

»Al  1000  mMi. 


Sourer  Bod  and  Jamiion  (1973).  ind  lUttloton.  Papotti.  and  Micheli  (1974). 


Jinn? 


run 


In  addition  to  the  line  rates,  "conditioning"  and  "termination" 
charges  of  approximately  $90  per  line  per  month  are  levied.  On 
the  basis  of  this  schedule,  it  can  be  seen  that  close  to  trans- 
continental distances  of  greater  than  2,000  miles  are  required 
before  the  average  rate  approaches  one  dollar  per  mile.  Only 
one  paper  (Ball  and  Jamison,  1973)  addressed  the  cost  of  short- 
line communications . These  rates  were  characterized  as  a con- 
stant function  of  mileage  that  approximates  $A  per  mile  per 
month  plus  line  conditioning  and  termination  charges  of  approxi- 
mately $125  per  month. 

Ball  and  JamiBon,  (1973)  was  also  the  only  paper  that 
addressed  the  costs  of  equipment  associated  with  land-line 
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communications.  Since  the  document  was  not  concerned  with  par- 
ticular hardware  systems,  it  provided  only  typical  costs  for 
generic  types  of  equipments  with  no  references  to  sources  or 
particular  equipments  on  which  the  estimates  were  based.  The 
limits  of  these  estimates  are  shown  in  Table  E-18. 

TABLE  E-18.  COSTS  OF  LAND-LINE 
COMMUNICATIONS  EQUIPMENT 


■ ■ — 

Equipment 

Unit  Cost 
(Dol lars ) 

Mul t i pi exors 

8 Channel 

$ 1,600 

Centra  1 

7 , 000 

System 

10,000 

Modems 

1200  Baud 

500 

2400  Baud 

1 ,750 

4800  Baud 

5,400 

Multiplex  Computer 

9,000 

E.10  PROGRAM  DELIVERY:  MATERIALS 

No  data  were  found  for  these  costs  of  instruction. 

E.ll  PROGRAM  DELIVERY:  FACILITIES 

Furnished  instructional  facilities  are  a requirement  of  any 
formal  training  program  and,  given  a static-sized  force,  it  may 
be  considered  that  existing  facilities  would  generally  be  avail- 
able without  further  expenditure  for  commonly  used  assets.  How- 
ever, the  introduction  of  self-pacing  imposes  a requirement  for 
assets  not  associated  with  traditional  instruction,  and  the 
introduction  of  PLATO  IV  (or  any  other  computer-based  system) 
imposes  other  requirements.  These  are  properly  costs  of  the 


E-36 


newly  employed  training  technology,  i.e.t  CAI  or  CMI . Two  USAF 
PLATO  IV  demonstration  programs — Dallman,  DeLeo,  et  al.  , 1977, 
and  Steinkerchner , Delgnan,  Waters,  and  DeLeo,  1977,  reported 
costs  of  facility  modifications  and  furnishings  required  for 
Instruction . 

In  the  case  of  these  two  USAF  demonstrations,  only  two 
facility  items  fell  into  this  class.  The  first  is  individual 
learning  carrels  and  the  second  is  the  provision  of  electric  and 
pneumatic  outlets  at  the  carrels  to  service  the  PLATO  IV  termi- 
nals. Other  requirements  such  as  communication  links  would 
appear  to  be  required  also,  but  these  were  not  listed.  The  cost 
of  carrels  was  reported  at  $90  per  unit  in  one  study  and  at  $260 
per  unit  in  the  other.  No  descriptions  were  provided  to  explain 
such  a wide  difference,  but  it  is  of  note  that  Hess  and  Kantar, 
(1977)  reported  an  average  or  typical  cost  of  carrels  for  pro- 
grammed instruction  at  $160 . 

The  cost  of  installing  electric  and  pneumatic  service  can 
be  expected  to  vary  widely  as  a function  of  the  characteristics 
of  the  building  in  which  they  are  installed.  In  the  case  of 
these  two  programs,  service  for  30  carrels  was  installed  at  an 
average  cost  of  $6l  per  carrel  and  service  for  20  carrels  was 
installed  at  an  average  of  $141  per  carrel.  In  neither  case  was 
a breakdown  between  electric  and.  pneumatic  provided,  and  in 
neither  case  was  it  noted  whether  pneumatic  service  was  already 
available  in  the  building  or  whether  the  program  involved  pro- 
curement and  installation  of  a compressor  as  well  as  the  running 
of  the  compressed  air  lines. 

E . 1 2 PROGRAM  MANAGEMENT  AND  ADMINISTRATION 

No  data  were  found  on  these  costs  of  instruction. 


E . 1 3 STUDENT  PERSONNEL 


Data  on  costs  of  Student  Personnel  are  Included  In  Section 
E.5,  Program  Delivery:  Instruction,  Including  Instructors  and 
Instruction  Support  Personnel  and  Student  Personnel. 


